[tc] Questions for TC Candidates
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
The questions follow. Don't feel obliged to answer all of these. The point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
* How do you account for the low number of candidates? Do you consider this a problem? Why or why not?
* Compare and contrast the role of the TC now to 4 years ago. If you weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
* What, to you, is the single most important thing the OpenStack community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
* If you had a magic wand and could inspire and make a single sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
* What can the TC do to make sure that the community (in its many dimensions) is informed of and engaged in the discussions and decisions of the TC?
* How do you counter people who assert the TC is not relevant? (Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
That's probably more than enough. Thanks for your attention.
Hi Chris,
Thanks for kicking this off. I've added my replies in-line.
Thank you for your past term as well.
Regards, Mohammed
On Wed, Feb 20, 2019 at 9:49 AM Chris Dent cdent+os@anticdent.org wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
The questions follow. Don't feel obliged to answer all of these. The point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
- How do you account for the low number of candidates? Do you consider this a problem? Why or why not?
Just for context, I wanted to share the following numbers to formulate my response:
Ocata candidates: 21 Pike candidates: 14 Queens candidates: 16 Rocky candidates: 10
We're indeed seeing the numbers grow cycle over cycle. However, a lot of the candidates are people that most seem to have ran once and upon not being elected, they didn't take a chance to go again. I think perhaps we should encourage reaching out to those previous candidates, especially those who are still parts of the community still to nominate themselves again.
I do however think that with the fact that our software is becoming more stable and having less overall contributors than before, it might be a good time to evaluate the size of the TC, but that could be a really interesting challenge to deal with and I'm not quite so sure yet about how we can approach that.
I don't think it's a problem, we had a really quiet start but then a lot of people put their names in. I think if the first candidate had come in a bit earlier, we would have seen more candidates because I get this feeling no one wants to go "first".
- Compare and contrast the role of the TC now to 4 years ago. If you weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
4 years ago, we were probably around the Kilo release cycle at that time and things were a lot different in the ecosystem. At the time, I think the TC had more of a role of governing as the projects had plenty of traction and things were moving.
As OpenStack seems to come closer to delivering most of the value that you need, without needing as much effort, I think it's important for us to try and envision how we can better place OpenStack in the overall infrastructure ecosystem and focus on marketing it.
I speak a lot to users and deployers daily and I find out a lot of things about current impressions of OpenStack, once I explain it to them, they are all super impressed by it so I think we need to do a better job at educating people.
Also, I think the APAC region is one that is a big growing user and community of OpenStack that we usually don't put as much thought into. We need to make sure that we invest more time into the community there.
- What, to you, is the single most important thing the OpenStack community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
I think our tooling is hard to use. I really love it, but it's really not straightforward for most new comers.
The majority of users are familiar with the GitHub workflow, the Gerrit one is definitely one that needs a bit of a learning curve. I think this introduces a really weird situation where if I'm not familiar with all of that and I want to submit a patch that's a simple change, it will take me more work to get setup on Gerrit than it does to make the fix.
I think most people give up and just don't want to bother at that point, perhaps a few more might be more inclined to get through it but it's really a lot of work to allow pushing a simple patch.
- If you had a magic wand and could inspire and make a single sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
Oh.
- Stop using RabbitMQ as an RPC, it's the worst most painful component to run in an entire OpenStack deployment. It's always broken. Switch into something that uses HTTP + service registration to find endpoints.
- Use Keystone as an authoritative service catalog, stop having to configure URLs for services inside configuration files. It's confusing and unreliable and causes a lot of breakages often.
- SSL first. For most services, the overhead is so small, I don't see why we wouldn't ever have all services to run SSL only.
- Single unified client, we're already moving towards this with the OpenStack client but it's probably been one of our biggest weaknesses that have not been completed and fully cleared out.
Those are a few that come to mind right now, I'm sure I could come up with so much more.
- What can the TC do to make sure that the community (in its many dimensions) is informed of and engaged in the discussions and decisions of the TC?
We need to follow the mailing lists and keep up to date at what users are trying to use OpenStack for. There's emerging use cases such as using it for edge deployments, the increase of bare-metal deployments (ironic) and thinking about how it can benefit end users, all of this can be seen by following mailing list discussions, Twitter-verse, and other avenues.
I've also found amazing value in being part of WeChat communities which bring a lot of insight from the APAC region.
- How do you counter people who assert the TC is not relevant? (Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
This is a tough one. That's something we need to work and change, I think that historically the involvement of the TC and projects have been very hands-off because of the velocity that projects moved at.
Now that we're a bit slower, I think that having the TC involved in the projects can be very interesting. It provides access to a group of diverse and highly technical individuals from different backgrounds (operators, developers -- but maybe not as much users) to chime in on certain directions of the projects.
That's probably more than enough. Thanks for your attention.
Thank you for starting this.
-- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
On 2/20/2019 9:24 AM, Mohammed Naser wrote:
- Use Keystone as an authoritative service catalog, stop having to configure URLs for services inside configuration files. It's confusing and unreliable and causes a lot of breakages often.
Nova did this in pike and queens [1] and is probably a good "show and tell" kind of thing that could be done as a cross-project community goal, or at least for the projects that depend on other openstack services. I think the one thing nova hasn't done this with yet is cinder [2].
[1] https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/us... [2] https://review.openstack.org/#/c/508345/
On Wed, 2019-02-20 at 10:24 -0500, Mohammed Naser wrote:
Hi Chris,
Thanks for kicking this off. I've added my replies in-line.
Thank you for your past term as well.
Regards, Mohammed
On Wed, Feb 20, 2019 at 9:49 AM Chris Dent cdent+os@anticdent.org wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
The questions follow. Don't feel obliged to answer all of these. The point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
- How do you account for the low number of candidates? Do you consider this a problem? Why or why not?
Just for context, I wanted to share the following numbers to formulate my response:
Ocata candidates: 21 Pike candidates: 14 Queens candidates: 16 Rocky candidates: 10
We're indeed seeing the numbers grow cycle over cycle. However, a lot of the candidates are people that most seem to have ran once and upon not being elected, they didn't take a chance to go again. I think perhaps we should encourage reaching out to those previous candidates, especially those who are still parts of the community still to nominate themselves again.
I do however think that with the fact that our software is becoming more stable and having less overall contributors than before, it might be a good time to evaluate the size of the TC, but that could be a really interesting challenge to deal with and I'm not quite so sure yet about how we can approach that.
I don't think it's a problem, we had a really quiet start but then a lot of people put their names in. I think if the first candidate had come in a bit earlier, we would have seen more candidates because I get this feeling no one wants to go "first".
- Compare and contrast the role of the TC now to 4 years ago. If you weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
4 years ago, we were probably around the Kilo release cycle at that time and things were a lot different in the ecosystem. At the time, I think the TC had more of a role of governing as the projects had plenty of traction and things were moving.
As OpenStack seems to come closer to delivering most of the value that you need, without needing as much effort, I think it's important for us to try and envision how we can better place OpenStack in the overall infrastructure ecosystem and focus on marketing it.
I speak a lot to users and deployers daily and I find out a lot of things about current impressions of OpenStack, once I explain it to them, they are all super impressed by it so I think we need to do a better job at educating people.
Also, I think the APAC region is one that is a big growing user and community of OpenStack that we usually don't put as much thought into. We need to make sure that we invest more time into the community there.
- What, to you, is the single most important thing the OpenStack community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
I think our tooling is hard to use. I really love it, but it's really not straightforward for most new comers.
The majority of users are familiar with the GitHub workflow, the Gerrit one is definitely one that needs a bit of a learning curve. I think this introduces a really weird situation where if I'm not familiar with all of that and I want to submit a patch that's a simple change, it will take me more work to get setup on Gerrit than it does to make the fix.
I think most people give up and just don't want to bother at that point, perhaps a few more might be more inclined to get through it but it's really a lot of work to allow pushing a simple patch.
- If you had a magic wand and could inspire and make a single sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
Oh.
- Stop using RabbitMQ as an RPC, it's the worst most painful component to run in an entire OpenStack deployment. It's always broken. Switch into something that uses HTTP + service registration to find endpoints.
as an on looker i have mixed feeling about this statement. RabbitMQ can have issue at scale but it morstly works when its not on fire. Would you be advocating building a openstack specific RPC layer perhaps using keystone as the service registry and a custom http mechanism or adopting an existing technology like grpc?
investigating an alternative RPC backend has come up in the past (zeromq and qupid) and i think it has merit but im not sure as a comunity creating a new RPC framework is a project wide effort that openstack need to solve. that said zaqar is a thing https://docs.openstack.org/zaqar/latest/ if it is good enough for our endusers to consume perhaps it would be up to the task of being openstack rpc transport layer.
anyway my main question was would you advocate adoption of an exisiting technology or creating our own solution if we were to work on this goal as a community.
Use Keystone as an authoritative service catalog, stop having to configure URLs for services inside configuration files. It's confusing and unreliable and causes a lot of breakages often.
SSL first. For most services, the overhead is so small, I don't see why we wouldn't ever have all services to run SSL only.
Single unified client, we're already moving towards this with the OpenStack client but it's probably been one of our biggest weaknesses that have not been completed and fully cleared out.
Those are a few that come to mind right now, I'm sure I could come up with so much more.
- What can the TC do to make sure that the community (in its many dimensions) is informed of and engaged in the discussions and decisions of the TC?
We need to follow the mailing lists and keep up to date at what users are trying to use OpenStack for. There's emerging use cases such as using it for edge deployments, the increase of bare-metal deployments (ironic) and thinking about how it can benefit end users, all of this can be seen by following mailing list discussions, Twitter-verse, and other avenues.
I've also found amazing value in being part of WeChat communities which bring a lot of insight from the APAC region.
- How do you counter people who assert the TC is not relevant? (Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
This is a tough one. That's something we need to work and change, I think that historically the involvement of the TC and projects have been very hands-off because of the velocity that projects moved at.
Now that we're a bit slower, I think that having the TC involved in the projects can be very interesting. It provides access to a group of diverse and highly technical individuals from different backgrounds (operators, developers -- but maybe not as much users) to chime in on certain directions of the projects.
That's probably more than enough. Thanks for your attention.
Thank you for starting this.
-- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
Hi Chris,
Thanks for getting us started :) replies inline below.
- How do you account for the low number of candidates? Do you
consider this a problem? Why or why not?
Change is inevitable, and in the last 3 years (distinctly since the Boston summit) there have been massive changes to our community. OpenStack went from being the new hotness to becoming a stable, secure open source project that is well respected in the open source community and that was reflected by the amount of people contributing to the projects.
I see the low number of candidates for the Train TC election as a direct reflection these changes.
While a position on the TC still garners a high level of respect from peers and employers, I have seen a distinct decline in the push for leadership positions from large-scale investors, due to OpenStack's stability and the resulting decline for large changes to be made to OpenStack as a product.
Is this a problem? No. Not if we view this as a new start. Defining who are are now as a stable open source product will set us apart from communities that wish to continue to ride the success high.
- Compare and contrast the role of the TC now to 4 years ago. If you
weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
The TC has always been a position of governance that is (debatably) clearly defined as an elected group that provides technical leadership for OpenStack as a whole. I believe it is not the question of what the TC is, but rather, what is the "technical guidance" that the TC provides, and how that has evolved.
Four (4) years ago, OpenStack was in a different place in the product life cycle. The hype was high, and we had buy-in from a wide range of investors coming from all different parts of the technology industry.
My experience with the TC then was more of a "governing" body helping to shape an incredibly fast growing community and product, whereas now I see it more as a mediation, communication, and community platform that has a focus on technical issues.
- What, to you, is the single most important thing the OpenStack
community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
As my experience stems directly from documentation, I genuinely believe that this is part of the ticket. Speaking as a communicator and a collaborator, I believe we have along the way lost touch with defining a minimum barrier to entry.
When I started in 2014, I was not particularly technically minded. I had worked at Red Hat for 2 years, and my experience had enabled me to understand "Cloud" and XML. I was hired by Rackspace with the proviso that, "We need good writers, we can teach you the technology." As a result, (and with some help) I found it easy and accessible to begin contributing and I've been here since.
To be able to build documentation today, as a new comer, I would need to install package dependencies for each repository. I have to read at least 2 different contributor guides to get started. And we still are yet to really open our world to our Windows user friends. Installing a package dep isn't hard or it is time consuming, but understanding and knowing what they. I believe we need to be mindful of the time people have. Whether or not they are working with OpenStack as a part of their employment, or if it is in their spare time.
- What can the TC do to make sure that the community (in its many
dimensions) is informed of and engaged in the discussions and decisions of the TC?
This is tough, because there are already so many ways that the TC engages with community and I think that's brilliant. The strong presence on the discuss ML, the ability to join the IRC channels, and the genuine interest each TC member has in open discussion.
By most standards, this is an incredibly active and public group and I do not wish to criticise it, only encourage what we have - and if new ways of communicating are requested, that we actively seek adoption to ensure we are including everyone.
- How do you counter people who assert the TC is not relevant?
(Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
This relates to my answer to your first question, OpenStack has undergone massive changes not only in community but in focus. It is common that when things change, governing bodies fail to change quickly enough. I can understand how people would come to this conclusion.
However, the TC is changing and that is evident by the large number of TC members who stood down this election and the number of those that have elected to stand for Train. I find the challenge of ensuring that the TC remains relevant to be an important part of what I stand for as a candidate. This means adapting to more changes in the future.
That's probably more than enough. Thanks for your attention.
I'd say that's probably more than enough from me too. Thanks for your questions, hopefully my answers are equally insightful :)
Cheers,
Alex
Thanks Chris for asking us questions so we can clarify our opinions.
On Wed, Feb 20, 2019 at 3:52 PM Chris Dent cdent+os@anticdent.org wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
I agree with you on this point. It's important for OpenStack to have time to discuss about mandates.
The questions follow. Don't feel obliged to answer all of these. The
point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
- How do you account for the low number of candidates? Do you consider this a problem? Why or why not?
Yes, again, I agree and to be honest, when I only saw we were only having 4 candidates 8 hours before the deadline, I said to myself "OK, you love OpenStack. You think the TC is important. But then, why aren't you then throwing your hat ?" We all have opinions, right ? But then, why people don't want to be in the TC ? Because we don't have a lot of time for it ? Or because people think the TC isn't important ?
I don't want to discuss about politics here. But I somehow see a parallel in between what the TC is and what the European Union is : both are governances not fully decision-makers but are there for sharing same rules and vision. If we stop having the TC, what would become OpenStack ? Just a set of parallel projects with no common guidance ?
The fact that a large number of candidacies went very late (including me) is a bit concerning to me. How can we become better ? I have no idea but saying that probably given the time investment it requires, most of the candidacies were probably holding some management acceptance before people would propose their names. Probably worth thinking about how the investment it requires, in particular given we have less full-time contributors that can dedicate large time for governance.
* Compare and contrast the role of the TC now to 4 years ago. If you
weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
4 years ago, we were in the Kilo timeframe. That's fun you mention this period, because at that exact time of the year, the TC voted on one of the probably most important decisions that impacted OpenStack : The Big Tent reform [1] Taking a look at this time, I remember frustration and hard talks but also people committed to change things. This decision hasn't changed a lot the existing service projects that were before the Big Tent, but it actually created a whole new ecosystem for developers. It had challenges but it never required to be abandoned, which means the program is a success.
Now the buzz is gone and the number of projects stable, the TC necessarly has to mutate to a role of making sure all the projects sustain the same pace and reliability. Most of the challenges for the TC is now about defining and applying criterias for ensuring that all our projects have a reasonable state for production. If you see my candidacy letter, two of my main drivers for my nomination are about upgradability and scalability concerns.
* What, to you, is the single most important thing the OpenStack
community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
There are two very distinct reasons when a company decides to downstream-only : either by choice or because of technical reasons. I don't think a lot of companies decide to manage technical debt on their own by choice. OpenStack is nearly 9 years old and most of the users know the price it is.
Consequently, I assume that the reasons are technical : 1/ they're running an old version and haven't upgraded (yet). We have good user stories of large cloud providers that invested in upgrades (for example OVH) and see the direct benefit of it. Maybe we can educate more on the benefits of upgrading frequently. 2/ they think upstreaming is difficult. I'm all open to hear the barriers they have. For what it's worth, OpenStack invested a lot in mentoring with the FirstContact SIG, documentation and Upstream Institute. There will probably also be a new program about peer-mentoring and recognition [2] if the community agrees with the idea. Honestly, I don't know what do do more. If you really can't upstream but care about your production, just take a service contract I guess.
- If you had a magic wand and could inspire and make a single sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
Take me as a fool but I don't think the role of the TC is to drive architectural decision between projects. The TC can help two projects to discuss, the TC can (somehow) help moderate between two teams about some architectural concern but certainly not be the driver of such change.
That doesn't mean the TC can't be technical. We have goals, for example. But in order to have well defined goals that are understandable by project contributors, we also need to have the projects be the drivers of such architectural changes.
- What can the TC do to make sure that the community (in its many dimensions) is informed of and engaged in the discussions and decisions of the TC?
You made a very good job in providing TC feedback. I surely think the TC has to make sure that a regular weekly feedback is provided. For decisions that impact projects, I don't really see how TC members can vote without getting feedback from the project contributors, so here I see communication (thru Gerrit at least).
- How do you counter people who assert the TC is not relevant? (Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
Again, I think that is a matter of considering the TC responsibilities. We somehow need to clarify what are those responsibilities and I think I voiced on that above.
That's probably more than enough. Thanks for your attention.
I totally appreciate you challenging us. That's very important that people vote based on opinions rather than popularity. -Sylvain
[1] https://governance.openstack.org/tc/resolutions/20141202-project-structure-r... [2] https://review.openstack.org/#/c/636956/
-- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
On 2/20/19 11:23 AM, Sylvain Bauza wrote:
Thanks Chris for asking us questions so we can clarify our opinions.
On Wed, Feb 20, 2019 at 3:52 PM Chris Dent <cdent+os@anticdent.org mailto:cdent%2Bos@anticdent.org> wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions. First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.I agree with you on this point. It's important for OpenStack to have time to discuss about mandates.
The questions follow. Don't feel obliged to answer all of these. The point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique! * How do you account for the low number of candidates? Do you consider this a problem? Why or why not?Yes, again, I agree and to be honest, when I only saw we were only having 4 candidates 8 hours before the deadline, I said to myself "OK, you love OpenStack. You think the TC is important. But then, why aren't you then throwing your hat ?" We all have opinions, right ? But then, why people don't want to be in the TC ? Because we don't have a lot of time for it ? Or because people think the TC isn't important ?
I don't want to discuss about politics here. But I somehow see a parallel in between what the TC is and what the European Union is : both are governances not fully decision-makers but are there for sharing same rules and vision. If we stop having the TC, what would become OpenStack ? Just a set of parallel projects with no common guidance ?
The fact that a large number of candidacies went very late (including me) is a bit concerning to me. How can we become better ? I have no idea but saying that probably given the time investment it requires, most of the candidacies were probably holding some management acceptance before people would propose their names. Probably worth thinking about how the investment it requires, in particular given we have less full-time contributors that can dedicate large time for governance.
* Compare and contrast the role of the TC now to 4 years ago. If you weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?4 years ago, we were in the Kilo timeframe. That's fun you mention this period, because at that exact time of the year, the TC voted on one of the probably most important decisions that impacted OpenStack : The Big Tent reform [1] Taking a look at this time, I remember frustration and hard talks but also people committed to change things. This decision hasn't changed a lot the existing service projects that were before the Big Tent, but it actually created a whole new ecosystem for developers. It had challenges but it never required to be abandoned, which means the program is a success.
Now the buzz is gone and the number of projects stable, the TC necessarly has to mutate to a role of making sure all the projects sustain the same pace and reliability. Most of the challenges for the TC is now about defining and applying criterias for ensuring that all our projects have a reasonable state for production. If you see my candidacy letter, two of my main drivers for my nomination are about upgradability and scalability concerns.
* What, to you, is the single most important thing the OpenStack community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?There are two very distinct reasons when a company decides to downstream-only : either by choice or because of technical reasons. I don't think a lot of companies decide to manage technical debt on their own by choice. OpenStack is nearly 9 years old and most of the users know the price it is.
Consequently, I assume that the reasons are technical : 1/ they're running an old version and haven't upgraded (yet). We have good user stories of large cloud providers that invested in upgrades (for example OVH) and see the direct benefit of it. Maybe we can educate more on the benefits of upgrading frequently. 2/ they think upstreaming is difficult. I'm all open to hear the barriers they have. For what it's worth, OpenStack invested a lot in mentoring with the FirstContact SIG, documentation and Upstream Institute. There will probably also be a new program about peer-mentoring and recognition [2] if the community agrees with the idea. Honestly, I don't know what do do more. If you really can't upstream but care about your production, just take a service contract I guess.
* If you had a magic wand and could inspire and make a single sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?Take me as a fool but I don't think the role of the TC is to drive architectural decision between projects. The TC can help two projects to discuss, the TC can (somehow) help moderate between two teams about some architectural concern but certainly not be the driver of such change.
Is there a particular reason why you feel this way?
I think the TC is in a great position to have a profound impact on the architecture of OpenStack, with a caveat.
I believe if you ask anyone with even a brief history in OpenStack, you'll dust up some architectural opinions. For example, Jim and Mohammed have already pointed out a bunch in their responses. Another example, Melanie and I had a productive discussion today about how restructuring the architecture of policy enforcement could significantly improvement usability and security [0], which certainly isn't specific to keystone or nova. I don't think we have to look very far to find excellent areas for improvement. As others have noted, the project is at a point where development and hype isn't nearly as intense as it was 4 years ago. While contributor numbers, in a way, reflect project stabilization, I also think it puts us in a prime position to address some of the architectural pain points we've grown to live with over the years. I think we can use the opportunity to make services more consistent, giving consumers and users a more refined and polished experience, among other benefits.
That said, I certainly think if the TC is to _facilitate_ in architectural decisions, it needs to be done in the open and with plenty of communication and feedback with the entire community. Similar to the approach we try and take with community goals.
I understand there may be a fine line in making decisions of this nature at the TC level, but I also think it presents numerous opportunities to communicate and focus efforts in a unified direction. I see that involvement range from socializing issues to advocating for sponsorship on a particular initiative to diving into the problem and helping projects directly.
[0] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-ke...
That doesn't mean the TC can't be technical. We have goals, for example. But in order to have well defined goals that are understandable by project contributors, we also need to have the projects be the drivers of such architectural changes.
* What can the TC do to make sure that the community (in its many dimensions) is informed of and engaged in the discussions and decisions of the TC?You made a very good job in providing TC feedback. I surely think the TC has to make sure that a regular weekly feedback is provided. For decisions that impact projects, I don't really see how TC members can vote without getting feedback from the project contributors, so here I see communication (thru Gerrit at least).
* How do you counter people who assert the TC is not relevant? (Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)Again, I think that is a matter of considering the TC responsibilities. We somehow need to clarify what are those responsibilities and I think I voiced on that above.
That's probably more than enough. Thanks for your attention.I totally appreciate you challenging us. That's very important that people vote based on opinions rather than popularity. -Sylvain
[1] https://governance.openstack.org/tc/resolutions/20141202-project-structure-r... [2] https://review.openstack.org/#/c/636956/
-- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
On Thu, Feb 21, 2019 at 1:54 AM Lance Bragstad lbragstad@gmail.com wrote:
On 2/20/19 11:23 AM, Sylvain Bauza wrote:
Thanks Chris for asking us questions so we can clarify our opinions.
On Wed, Feb 20, 2019 at 3:52 PM Chris Dent cdent+os@anticdent.org wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
I agree with you on this point. It's important for OpenStack to have time to discuss about mandates.
The questions follow. Don't feel obliged to answer all of these. The
point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
- How do you account for the low number of candidates? Do you consider this a problem? Why or why not?
Yes, again, I agree and to be honest, when I only saw we were only having 4 candidates 8 hours before the deadline, I said to myself "OK, you love OpenStack. You think the TC is important. But then, why aren't you then throwing your hat ?" We all have opinions, right ? But then, why people don't want to be in the TC ? Because we don't have a lot of time for it ? Or because people think the TC isn't important ?
I don't want to discuss about politics here. But I somehow see a parallel in between what the TC is and what the European Union is : both are governances not fully decision-makers but are there for sharing same rules and vision. If we stop having the TC, what would become OpenStack ? Just a set of parallel projects with no common guidance ?
The fact that a large number of candidacies went very late (including me) is a bit concerning to me. How can we become better ? I have no idea but saying that probably given the time investment it requires, most of the candidacies were probably holding some management acceptance before people would propose their names. Probably worth thinking about how the investment it requires, in particular given we have less full-time contributors that can dedicate large time for governance.
- Compare and contrast the role of the TC now to 4 years ago. If you
weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
4 years ago, we were in the Kilo timeframe. That's fun you mention this period, because at that exact time of the year, the TC voted on one of the probably most important decisions that impacted OpenStack : The Big Tent reform [1] Taking a look at this time, I remember frustration and hard talks but also people committed to change things. This decision hasn't changed a lot the existing service projects that were before the Big Tent, but it actually created a whole new ecosystem for developers. It had challenges but it never required to be abandoned, which means the program is a success.
Now the buzz is gone and the number of projects stable, the TC necessarly has to mutate to a role of making sure all the projects sustain the same pace and reliability. Most of the challenges for the TC is now about defining and applying criterias for ensuring that all our projects have a reasonable state for production. If you see my candidacy letter, two of my main drivers for my nomination are about upgradability and scalability concerns.
- What, to you, is the single most important thing the OpenStack
community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
There are two very distinct reasons when a company decides to downstream-only : either by choice or because of technical reasons. I don't think a lot of companies decide to manage technical debt on their own by choice. OpenStack is nearly 9 years old and most of the users know the price it is.
Consequently, I assume that the reasons are technical : 1/ they're running an old version and haven't upgraded (yet). We have good user stories of large cloud providers that invested in upgrades (for example OVH) and see the direct benefit of it. Maybe we can educate more on the benefits of upgrading frequently. 2/ they think upstreaming is difficult. I'm all open to hear the barriers they have. For what it's worth, OpenStack invested a lot in mentoring with the FirstContact SIG, documentation and Upstream Institute. There will probably also be a new program about peer-mentoring and recognition [2] if the community agrees with the idea. Honestly, I don't know what do do more. If you really can't upstream but care about your production, just take a service contract I guess.
- If you had a magic wand and could inspire and make a single sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
Take me as a fool but I don't think the role of the TC is to drive architectural decision between projects. The TC can help two projects to discuss, the TC can (somehow) help moderate between two teams about some architectural concern but certainly not be the driver of such change.
Is there a particular reason why you feel this way?
I think the TC is in a great position to have a profound impact on the architecture of OpenStack, with a caveat.
I believe if you ask anyone with even a brief history in OpenStack, you'll dust up some architectural opinions. For example, Jim and Mohammed have already pointed out a bunch in their responses. Another example, Melanie and I had a productive discussion today about how restructuring the architecture of policy enforcement could significantly improvement usability and security [0], which certainly isn't specific to keystone or nova. I don't think we have to look very far to find excellent areas for improvement. As others have noted, the project is at a point where development and hype isn't nearly as intense as it was 4 years ago. While contributor numbers, in a way, reflect project stabilization, I also think it puts us in a prime position to address some of the architectural pain points we've grown to live with over the years. I think we can use the opportunity to make services more consistent, giving consumers and users a more refined and polished experience, among other benefits.
That said, I certainly think if the TC is to _facilitate_ in architectural decisions, it needs to be done in the open and with plenty of communication and feedback with the entire community. Similar to the approach we try and take with community goals.
I understand there may be a fine line in making decisions of this nature at the TC level, but I also think it presents numerous opportunities to communicate and focus efforts in a unified direction. I see that involvement range from socializing issues to advocating for sponsorship on a particular initiative to diving into the problem and helping projects directly.
Hi Lance, Thanks for your reply and leaving me a chance to clarify my thoughts on a possible strawman. I'm not opposed to architectural modifications, and like you mention the fact that we're past the hype leaves us a good opportunity for revisiting some crucial cross-project pain points like the ones you mention (and which I'm fully onboard). For example, I remember some hard talks we had at Summits about a potential 'Architecture WG' and all the concerns that were around it.
What I said earlier was that IMHO the TC should not dictate any architectural changes or be the initiator of important architectural redesigns (unless I misunderstand the word 'driving' and in this case my bad). I rather prefer to see the TC to be a facilitator of such initiatives coming from individuals or projects (in particular given we're not in a position where we have less resources that can take time to work full-time on those big changes that are multi-cycle). We sometimes failed in the past to have such initiatives, but now we have goals we're in a better position than in the past. That said, given the nature of goals to be community-wide, I'm open for discussing other ways to engage architectural redesigns that imply only a few services (eg. placement and other services that would consume it). We generally have cross-project sessions at the PTG that help driving those discussions, but I somehow feel we miss some better way to promote those. architectural redesigns.
Hoping I answered your concerns.
-Sylvain
[0] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-ke...
That doesn't mean the TC can't be technical. We have goals, for example. But in order to have well defined goals that are understandable by project contributors, we also need to have the projects be the drivers of such architectural changes.
- What can the TC do to make sure that the community (in its many dimensions) is informed of and engaged in the discussions and decisions of the TC?
You made a very good job in providing TC feedback. I surely think the TC has to make sure that a regular weekly feedback is provided. For decisions that impact projects, I don't really see how TC members can vote without getting feedback from the project contributors, so here I see communication (thru Gerrit at least).
- How do you counter people who assert the TC is not relevant? (Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
Again, I think that is a matter of considering the TC responsibilities. We somehow need to clarify what are those responsibilities and I think I voiced on that above.
That's probably more than enough. Thanks for your attention.
I totally appreciate you challenging us. That's very important that people vote based on opinions rather than popularity. -Sylvain
[1] https://governance.openstack.org/tc/resolutions/20141202-project-structure-r... [2] https://review.openstack.org/#/c/636956/
-- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
On Wed, Feb 20, 2019 at 9:52 AM Chris Dent cdent+os@anticdent.org wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
The questions follow. Don't feel obliged to answer all of these. The point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
I considered top-posting just because you said this :P
- How do you account for the low number of candidates? Do you consider this a problem? Why or why not?
I suspect that this is a combination of a few things:
* A decline in contributors who have enough time dedicated to spend contributing to the TC. We're far down the hill of the hype cycle, and not as many people can get time from their employers to do the "softer" things in the community. I'm not sure if this is a problem or not - does the current TC feel overloaded with work? If not, maybe we don't need so many people.
* A decline in contributors who think being on the TC can help further their goals. Most contributors have focused technical goals, and being a part of the TC usually doesn't accelerate those. This seems fine to me, though I'd love to have more people with larger technical goals (more on this later).
* The change in the perceived role of the TC. I'll dig into this more in the next question.
* Compare and contrast the role of the TC now to 4 years ago. If you
weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
As Sylvain mentioned, this is near the time of the big tent reform. Until then, the TC was getting into technical details within/between projects. The big tent reform was an admission that the TC overseeing technical bits doesn't scale to something OpenStack-sized. As such, the role of the TC has become far less technical, instead becoming stewards of the technical community. In some ways, this is a good thing, as it gives projects autonomy to do what is right for their project's users.
However, this means that there are few people driving for larger OpenStack-wide changes to improve the experience for deployers, operators, and users. There are some awesome people (you know who you are) making smaller improvements that improve the story, but nothing like the architectural changes we need to really fix some of our underlying issues.
I believe the TC needs to drive more of these big-picture changes, rather than only focusing on governance. I'm not sure if that means doing the research to decide what to focus on, writing POCs and doing performance tests, doing the actual implementation, or just herding the right cats to get it done. I'm also unsure how much time TC members would be able to commit to this. But, I think without the TC driving things, it will never get done.
* What, to you, is the single most important thing the OpenStack
community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
Mohammed mentioned our tooling not being great here, and I agree. But we've also decided time and time again that's not changing, so. I think what the community needs to be doing is to be willing to spend the time mentoring these people, and holding their hand while they stumble through gerrit or writing complex tests. We should also be willing to take a patch from a contributor (whether by gerrit, email, or bar napkin), and finish it for them. For example, An operator that knows just enough python to find and fix an off-by-one error probably isn't going to be able to fix the unit tests or think through upgrade concerns.
I actually think we do a pretty good job with this today. Of course, it can always improve, so I'd like to see us continue that. As far as the TC's role, I think they should continue to encourage this behavior, and maybe make some sort of push to communicating the fact that we're willing to help outside of our usual channels. Busy users probably aren't reading this mailing list much - we should find some more accessible ways to call for these contributions.
- If you had a magic wand and could inspire and make a single sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
The fun question! Note: these are strong opinions, held loosely. If someone can prove that these changes won't improve OpenStack, I'm happy to drop them. :)
I agree with Mohammed, using rabbitmq for RPC isn't working out. I'd like us to be using HTTP and service discovery.
I also think that running more than one agent on a hypervisor isn't productive. These agents are fairly tightly coupled and are interacting with the same resources - we should combine them into a single agent which services talk to (over HTTP, of course). This should be organized under a single "compute node" or "hypervisor" team. This aligns the team more with a layer of the stack, rather than the APIs that abstracts those layers. Bonus points if this agent becomes easier to deploy - a container or a statically linked binary makes the hypervisor much easier to manage. Just image or PXE boot an OS, drop the binary in, and go.
Last, we need to fix the developer experience in OpenStack. In my experience, tooling that allows developers to iterate on changes quickly is the number one quality of life improvement a software project can do. Our services often take tens of minutes to run unit tests, and getting devstack up and running can easily take an hour. This is a huge turn-off for casual contributors, and a huge timesink for regular contributors.
As mentioned above, I believe that changes like this are fundamental to the future of OpenStack. We keep improving, but without fixing the underlying architectural issues, using or running OpenStack will always be painful. I believe that the TC needs to lead these initiatives, and continue to push on them, or else they won't get done.
- What can the TC do to make sure that the community (in its many dimensions) is informed of and engaged in the discussions and decisions of the TC?
Honestly, I don't believe that the average OpenStack community member really cares much about the discussions and decisions of the TC. Most of these don't directly affect said average person. See the next question.
One thing that the average community member does seem to care about is the goals process. I believe this is because these are technical changes which improve OpenStack as a whole. We should do more of that.
- How do you counter people who assert the TC is not relevant? (Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
Again, I don't think most of what the TC does is relevant to the average community member. I think that the TC needs to be more technically focused, and as such will be more relevant to the community. I hope to help steer us this way.
That's probably more than enough. Thanks for your attention.
Thanks for asking!
// jim
On 20/02/2019 14:46, Chris Dent wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
The questions follow. Don't feel obliged to answer all of these. The point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
- How do you account for the low number of candidates? Do you
consider this a problem? Why or why not?
I think we are reaching a more stable space, and the people who are developing the software are comfortable in the roles they are in.
As the demographic of our developers shifts east, our leadership is still very US / EU based, which may be why we are not getting the same amount of people growing into TC candidates.
- Compare and contrast the role of the TC now to 4 years ago. If you
weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
4 years ago, was just before the big tent I think? Ironically, there was a lot of the same discussion - python3, new project requirements (at that point the incubation requirements), asyncio / eventlet.
The TC was also in the process of dealing with a By-Laws change, in this case getting the trademark program off the ground.
We were still struggling with the "what is OpenStack?" question.
Looking back on the mailing list archives is actually quite interesting and while the topics are the same, a lot of the answers have changed.
- What, to you, is the single most important thing the OpenStack
community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
I think things like the review culture change have been good for this. The only other thing we can do is have more people reviewing, to make that first contact nice and quick, but E_NO_TIME or E_NO_HUMANS becomes the issue.
- If you had a magic wand and could inspire and make a single
sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
1: Single agent on each compute node that allows for plugins to do all the work required. (Nova / Neutron / Vitrage / watcher / etc)
2: Remove RMQ where it makes sense - e.g. for nova-api -> nova-compute using something like HTTP(S) would make a lot of sense.
3: Unified Error codes, with a central registry, but at the very least each time we raise an error, and it gets returned a user can see where in the code base it failed. e.g. a header that has OS-ERROR-COMPUTE-3142, which means that someone can google for something more informative than the VM failed scheduling
4: OpenTracing support in all projects.
5: Possibly something with pub / sub where each project can listen for events and not create something like designate did using notifications.
- What can the TC do to make sure that the community (in its many
dimensions) is informed of and engaged in the discussions and decisions of the TC?
This is a difficult question, especially in a community where a lot of contributors are sponsored.
The most effective way would be for the TC to start directly telling projects what to do - but I feel like that would mean that everyone would be unhappy with us.
- How do you counter people who assert the TC is not relevant?
(Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
Highlight the work done by the TC communicating with the board, guiding teams on what our vision is, and helping to pick goals. I think the goals are a great way, and we are starting to see the benifits as we continue with the practice.
For some people, we will always be surplus to requirements, and they just want to dig into bugs and features, and not worry about politics.
Thats fine - we just have to work with enough of the people on the teams to make sure that the project is heading in the correct direction, and as long as people can pick up what the priotities are from that process, I think we win.
That's probably more than enough. Thanks for your attention.
On Thu, Feb 21, 2019 at 6:14 PM Graham Hayes gr@ham.ie wrote:
On 20/02/2019 14:46, Chris Dent wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
The questions follow. Don't feel obliged to answer all of these. The point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
- How do you account for the low number of candidates? Do you consider this a problem? Why or why not?
I think we are reaching a more stable space, and the people who are developing the software are comfortable in the roles they are in.
As the demographic of our developers shifts east, our leadership is still very US / EU based, which may be why we are not getting the same amount of people growing into TC candidates.
- Compare and contrast the role of the TC now to 4 years ago. If you weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
4 years ago, was just before the big tent I think? Ironically, there was a lot of the same discussion - python3, new project requirements (at that point the incubation requirements), asyncio / eventlet.
The TC was also in the process of dealing with a By-Laws change, in this case getting the trademark program off the ground.
We were still struggling with the "what is OpenStack?" question.
Looking back on the mailing list archives is actually quite interesting and while the topics are the same, a lot of the answers have changed.
- What, to you, is the single most important thing the OpenStack community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
I think things like the review culture change have been good for this. The only other thing we can do is have more people reviewing, to make that first contact nice and quick, but E_NO_TIME or E_NO_HUMANS becomes the issue.
- If you had a magic wand and could inspire and make a single sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
1: Single agent on each compute node that allows for plugins to do all the work required. (Nova / Neutron / Vitrage / watcher / etc)
2: Remove RMQ where it makes sense - e.g. for nova-api -> nova-compute using something like HTTP(S) would make a lot of sense.
3: Unified Error codes, with a central registry, but at the very least each time we raise an error, and it gets returned a user can see where in the code base it failed. e.g. a header that has OS-ERROR-COMPUTE-3142, which means that someone can google for something more informative than the VM failed scheduling
4: OpenTracing support in all projects.
5: Possibly something with pub / sub where each project can listen for events and not create something like designate did using notifications.
That's the exact reason why I tried to avoid to answer about architectural changes I'd like to see it done. Because when I read the above lines, I'm far off any consensus on those. To answer 1. and 2. from my Nova developer's hat, I'd just say that we invented Cells v2 and Placement. To be clear, the redesign wasn't coming from any other sources but our users, complaining about scale. IMHO If we really want to see some comittee driving us about feature requests, this should be the UC and not the TC.
Whatever it is, at the end of the day, we're all paid by our sponsors. Meaning that any architectural redesign always hits the reality wall where you need to convince your respective Product Managers of the great benefit of the redesign. I'm maybe too pragmatic, but I remember so many discussions we had about redesigns that I now feel we just need hands, not ideas.
-Sylvain
- What can the TC do to make sure that the community (in its many
dimensions) is informed of and engaged in the discussions and decisions of the TC?
This is a difficult question, especially in a community where a lot of contributors are sponsored.
The most effective way would be for the TC to start directly telling projects what to do - but I feel like that would mean that everyone would be unhappy with us.
- How do you counter people who assert the TC is not relevant? (Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
Highlight the work done by the TC communicating with the board, guiding teams on what our vision is, and helping to pick goals. I think the goals are a great way, and we are starting to see the benifits as we continue with the practice.
For some people, we will always be surplus to requirements, and they just want to dig into bugs and features, and not worry about politics.
Thats fine - we just have to work with enough of the people on the teams to make sure that the project is heading in the correct direction, and as long as people can pick up what the priotities are from that process, I think we win.
That's probably more than enough. Thanks for your attention.
On 21/02/2019 17:28, Sylvain Bauza wrote:
On Thu, Feb 21, 2019 at 6:14 PM Graham Hayes <gr@ham.ie mailto:gr@ham.ie> wrote:
<snip>
> * If you had a magic wand and could inspire and make a single > sweeping architectural or software change across the services, > what would it be? For now, ignore legacy or upgrade concerns. > What role should the TC have in inspiring and driving such > changes? 1: Single agent on each compute node that allows for plugins to do all the work required. (Nova / Neutron / Vitrage / watcher / etc) 2: Remove RMQ where it makes sense - e.g. for nova-api -> nova-compute using something like HTTP(S) would make a lot of sense. 3: Unified Error codes, with a central registry, but at the very least each time we raise an error, and it gets returned a user can see where in the code base it failed. e.g. a header that has OS-ERROR-COMPUTE-3142, which means that someone can google for something more informative than the VM failed scheduling 4: OpenTracing support in all projects. 5: Possibly something with pub / sub where each project can listen for events and not create something like designate did using notifications.That's the exact reason why I tried to avoid to answer about architectural changes I'd like to see it done. Because when I read the above lines, I'm far off any consensus on those. To answer 1. and 2. from my Nova developer's hat, I'd just say that we invented Cells v2 and Placement.
Sure - this was if *I* had a magic wand - I have a completely different viewpoint to others. No community really ever has a full agreement.
From a TC perspective we have to look at these things from an overall view. My suggestions above were for *all* projects, specifically for #2 - I used a well known pattern as an example, but it can apply to Trove talking to DB instances, Octavia to LBaaS nodes (they already do this, and it is a good pattern), Zun, possibly Magnum (this is not an exaustive list, and may not suit all listed projects, I am taking them from the top of my head).
From what I understand there was even talk of doing it for Nova so that a central control plane could manage remote edge compute nodes without having to keep a RMQ connection alive across the WAN, but I am not sure where that got to.
To be clear, the redesign wasn't coming from any other sources but our users, complaining about scale. IMHO If we really want to see some comittee driving us about feature requests, this should be the UC and not the TC.
It should be a combination - UC and TC should be communicating about these requests - UC for the feedback, and the TC to see hwo they fit with the TCs vision for the direction of OpenStack.
Whatever it is, at the end of the day, we're all paid by our sponsors. Meaning that any architectural redesign always hits the reality wall where you need to convince your respective Product Managers of the great benefit of the redesign. I'm maybe too pragmatic, but I remember so many discussions we had about redesigns that I now feel we just need hands, not ideas.
I fully agree, and it has been an issue in the community for as long as I can remember. It doesn't mean that we should stop pushing the project forward. We have already moved the needle with the cycle goals, so we can influence what features are added to projects. Lets continue to do so.
<snip>
On Thu, Feb 21, 2019 at 6:47 PM Graham Hayes gr@ham.ie wrote:
On 21/02/2019 17:28, Sylvain Bauza wrote:
On Thu, Feb 21, 2019 at 6:14 PM Graham Hayes <gr@ham.ie mailto:gr@ham.ie> wrote:
<snip>
> * If you had a magic wand and could inspire and make a single > sweeping architectural or software change across the services, > what would it be? For now, ignore legacy or upgrade concerns. > What role should the TC have in inspiring and driving such > changes? 1: Single agent on each compute node that allows for plugins to do all the work required. (Nova / Neutron / Vitrage / watcher / etc) 2: Remove RMQ where it makes sense - e.g. for nova-api ->nova-compute
using something like HTTP(S) would make a lot of sense. 3: Unified Error codes, with a central registry, but at the veryleast
each time we raise an error, and it gets returned a user can see where in the code base it failed. e.g. a header that has OS-ERROR-COMPUTE-3142, which means that someone can google for something more informative than the VM failed scheduling 4: OpenTracing support in all projects. 5: Possibly something with pub / sub where each project can listenfor
events and not create something like designate did using notifications.That's the exact reason why I tried to avoid to answer about architectural changes I'd like to see it done. Because when I read the above lines, I'm far off any consensus on those. To answer 1. and 2. from my Nova developer's hat, I'd just say that we invented Cells v2 and Placement.
Sure - this was if *I* had a magic wand - I have a completely different viewpoint to others. No community really ever has a full agreement.
Fair point, we work with consensus, not full agreements. It's always good to keep that distinction in mind.
From a TC perspective we have to look at these things from an overall view. My suggestions above were for *all* projects, specifically for #2 - I used a well known pattern as an example, but it can apply to Trove talking to DB instances, Octavia to LBaaS nodes (they already do this, and it is a good pattern), Zun, possibly Magnum (this is not an exaustive list, and may not suit all listed projects, I am taking them from the top of my head).
I'd be interested in discussing the use cases requiring such important architectural splits. The main reason why Cells v2 was implemented was to address the MQ/DB scalability issue of 1000+ compute nodes. The Edge thingy came after this, so it wasn't the main driver for change. If the projects you mention have the same footprints at scale, then yeah I'm supportive of any redesign discussion that would come up.
That said, before stepping in into major redesigns, I'd wonder : could the inter-services communication be improved in terms of reducing payload ?
From what I understand there was even talk of doing it for Nova so that a central control plane could manage remote edge compute nodes without having to keep a RMQ connection alive across the WAN, but I am not sure where that got to.
That's a separate usecase (Edge) which wasn't the initial reason why we started implementing Cells V2. I haven't heard any request from the Edge WG during the PTGs about changing our messaging interface because $WAN but I'm open to ideas.
-Sylvain
To be clear, the redesign wasn't coming from any other sources but our
users, complaining about scale. IMHO If we really want to see some comittee driving us about feature requests, this should be the UC and not the TC.
It should be a combination - UC and TC should be communicating about these requests - UC for the feedback, and the TC to see hwo they fit with the TCs vision for the direction of OpenStack.
Whatever it is, at the end of the day, we're all paid by our sponsors. Meaning that any architectural redesign always hits the reality wall where you need to convince your respective Product Managers of the great benefit of the redesign. I'm maybe too pragmatic, but I remember so many discussions we had about redesigns that I now feel we just need hands, not ideas.
I fully agree, and it has been an issue in the community for as long as I can remember. It doesn't mean that we should stop pushing the project forward. We have already moved the needle with the cycle goals, so we can influence what features are added to projects. Lets continue to do so.
<snip>
On 21/02/2019 18:04, Sylvain Bauza wrote:
<snip>
I'd be interested in discussing the use cases requiring such important architectural splits. The main reason why Cells v2 was implemented was to address the MQ/DB scalability issue of 1000+ compute nodes. The Edge thingy came after this, so it wasn't the main driver for change. If the projects you mention have the same footprints at scale, then yeah I'm supportive of any redesign discussion that would come up.
That said, before stepping in into major redesigns, I'd wonder : could the inter-services communication be improved in terms of reducing payload ?
This is actually orthogonal to cells v2. There is other good reasons to remove RMQ in some places:
nova control plane <-> compute traffic can be point to point, so a HTTP request is perfectly workable for things that use calls() (cast() is a different story). This removes a lot of intermediate components, (oslo.messaging, RMQ, persistent connections, etc). It is not with out its own complexity, and potential pitfalls, but I am not going to design a spec on this thread :)
for other services using RMQ:
1. Having service VMs connect to RMQ means that if one VM gets compromised, the attacker could cause havoc on cloud by deleting VMs, networks, or other resources. You can help this by running multiple RMQ services, combinations or vhosts and permissions, but the service resources are still under threat in all cases.
2. Possibly having to open ports from in cloud workloads to the under cloud so that RMQ is accessible for the in cloud services.
This ties into the single agent for all openstack services - if we had a standard agent on machines that do things for openstack, we could have cross project TLS mutual auth, / app credentials / other auth tooling and do it once, and then just make sure that each image build script for in cloud services includes it.
From what I understand there was even talk of doing it for Nova so that a central control plane could manage remote edge compute nodes without having to keep a RMQ connection alive across the WAN, but I am not sure where that got to.That's a separate usecase (Edge) which wasn't the initial reason why we started implementing Cells V2. I haven't heard any request from the Edge WG during the PTGs about changing our messaging interface because $WAN but I'm open to ideas.
It was discussed with a few people from the Nova team in the Denver PTG Edge room from what I remember.
-Sylvain
> To be clear, the redesign wasn't coming from any other sources but our > users, complaining about scale. IMHO If we really want to see some > comittee driving us about feature requests, this should be the UC and > not the TC. It should be a combination - UC and TC should be communicating about these requests - UC for the feedback, and the TC to see hwo they fit with the TCs vision for the direction of OpenStack. > Whatever it is, at the end of the day, we're all paid by our sponsors. > Meaning that any architectural redesign always hits the reality wall > where you need to convince your respective Product Managers of the great > benefit of the redesign. I'm maybe too pragmatic, but I remember so many > discussions we had about redesigns that I now feel we just need hands, > not ideas. I fully agree, and it has been an issue in the community for as long as I can remember. It doesn't mean that we should stop pushing the project forward. We have already moved the needle with the cycle goals, so we can influence what features are added to projects. Lets continue to do so. <snip>
I think its good for the TC to discuss architectural shortcomings... Someone really needs to do it.
If the way to do that is to have folks be elected based on recommending architecture changes and we vote on electing them, at least thats some way for folks to provide feedback on whats important there. Gives us operators more of a way to provide feedback.
For example, I think CellsV2 mostly came about due to the current architecture not scaling well due to mysql/and torturing rabbit. The nova team felt I think it best to stick with the blessed architecture and try and scale it (not unreasonable from a single project perspective). But, rather then add a bunch of complexity which operators now suffer for, it could have been handled by fixing the underlying architectural issue. Stop torturing rabbit. If k8s can do 5000 nodes with one, non sharded control plane, nova should be able to too. Scheduling/starting a container and scheduling/starting a vm are not fundamentally different in their system requirements. Before, operators didn't have a frame of reference so just went along with it. Now they have more options and can more easily see the pain points in OpenStack and can decide to shift workload elsewhere. A single project can't make these sorts of overarching architectural decisions. The TC should do one of decide/help decide/facilitate deciding/delegate deciding. But someone needs to drive it, otherwise it gets dropped. That should be the TC IMO.
The TC candidates are talking more and more about OpenStack being stable. One development quote I like, "the code is done, not when there is nothing more to add, but nothing more to remove" speaks to me here... Do TC candidates think that should that be an architectural goal coming up soon? Figure out how to continue to do what OpenStack does, but do it simpler and/or with less code/services? That may require braking down some project walls. Is that a good thing to do?
Thanks, Kevin
________________________________ From: Sylvain Bauza [sbauza@redhat.com] Sent: Thursday, February 21, 2019 9:28 AM To: Graham Hayes Cc: openstack-discuss@lists.openstack.org Subject: Re: [tc] Questions for TC Candidates
On Thu, Feb 21, 2019 at 6:14 PM Graham Hayes <gr@ham.iemailto:gr@ham.ie> wrote: On 20/02/2019 14:46, Chris Dent wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
The questions follow. Don't feel obliged to answer all of these. The point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
- How do you account for the low number of candidates? Do you consider this a problem? Why or why not?
I think we are reaching a more stable space, and the people who are developing the software are comfortable in the roles they are in.
As the demographic of our developers shifts east, our leadership is still very US / EU based, which may be why we are not getting the same amount of people growing into TC candidates.
- Compare and contrast the role of the TC now to 4 years ago. If you weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
4 years ago, was just before the big tent I think? Ironically, there was a lot of the same discussion - python3, new project requirements (at that point the incubation requirements), asyncio / eventlet.
The TC was also in the process of dealing with a By-Laws change, in this case getting the trademark program off the ground.
We were still struggling with the "what is OpenStack?" question.
Looking back on the mailing list archives is actually quite interesting and while the topics are the same, a lot of the answers have changed.
- What, to you, is the single most important thing the OpenStack community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
I think things like the review culture change have been good for this. The only other thing we can do is have more people reviewing, to make that first contact nice and quick, but E_NO_TIME or E_NO_HUMANS becomes the issue.
- If you had a magic wand and could inspire and make a single sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
1: Single agent on each compute node that allows for plugins to do all the work required. (Nova / Neutron / Vitrage / watcher / etc)
2: Remove RMQ where it makes sense - e.g. for nova-api -> nova-compute using something like HTTP(S) would make a lot of sense.
3: Unified Error codes, with a central registry, but at the very least each time we raise an error, and it gets returned a user can see where in the code base it failed. e.g. a header that has OS-ERROR-COMPUTE-3142, which means that someone can google for something more informative than the VM failed scheduling
4: OpenTracing support in all projects.
5: Possibly something with pub / sub where each project can listen for events and not create something like designate did using notifications.
That's the exact reason why I tried to avoid to answer about architectural changes I'd like to see it done. Because when I read the above lines, I'm far off any consensus on those. To answer 1. and 2. from my Nova developer's hat, I'd just say that we invented Cells v2 and Placement. To be clear, the redesign wasn't coming from any other sources but our users, complaining about scale. IMHO If we really want to see some comittee driving us about feature requests, this should be the UC and not the TC.
Whatever it is, at the end of the day, we're all paid by our sponsors. Meaning that any architectural redesign always hits the reality wall where you need to convince your respective Product Managers of the great benefit of the redesign. I'm maybe too pragmatic, but I remember so many discussions we had about redesigns that I now feel we just need hands, not ideas.
-Sylvain
- What can the TC do to make sure that the community (in its many dimensions) is informed of and engaged in the discussions and decisions of the TC?
This is a difficult question, especially in a community where a lot of contributors are sponsored.
The most effective way would be for the TC to start directly telling projects what to do - but I feel like that would mean that everyone would be unhappy with us.
- How do you counter people who assert the TC is not relevant? (Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
Highlight the work done by the TC communicating with the board, guiding teams on what our vision is, and helping to pick goals. I think the goals are a great way, and we are starting to see the benifits as we continue with the practice.
For some people, we will always be surplus to requirements, and they just want to dig into bugs and features, and not worry about politics.
Thats fine - we just have to work with enough of the people on the teams to make sure that the project is heading in the correct direction, and as long as people can pick up what the priotities are from that process, I think we win.
That's probably more than enough. Thanks for your attention.
On Thu, Feb 21, 2019 at 7:28 PM Fox, Kevin M Kevin.Fox@pnnl.gov wrote:
I think its good for the TC to discuss architectural shortcomings... Someone really needs to do it.
If the way to do that is to have folks be elected based on recommending architecture changes and we vote on electing them, at least thats some way for folks to provide feedback on whats important there. Gives us operators more of a way to provide feedback.
For example, I think CellsV2 mostly came about due to the current architecture not scaling well due to mysql/and torturing rabbit. The nova team felt I think it best to stick with the blessed architecture and try and scale it (not unreasonable from a single project perspective). But, rather then add a bunch of complexity which operators now suffer for, it could have been handled by fixing the underlying architectural issue.
That's a bit unfortunate if you feel having more operator complexity with Cells V2 (rather than, per say, Cells V1) because Cells V2 is the default Nova now. Operators don't have to configure anything in order to start with a single cell, right ? Doing Cells V2 was actually what you said "fixing the underlying architectural issue". See https://docs.openstack.org/nova/latest/user/cells.html#manifesto for details.
Stop torturing rabbit. If k8s can do 5000 nodes with one, non sharded
control plane, nova should be able to too. Scheduling/starting a container and scheduling/starting a vm are not fundamentally different in their system requirements.
From a 30K-feet high level, yes. Also, I can point some recommended
architecture that allows you to have a single MQ for 5000 compute nodes with Nova, of course. But let's not jump into technical details here. We'll have the Forum late April which is the perfect place for operator/developer communication and address this particular concern. Feel free to propose a Forum session about this, I'd be happy to participate to it.
Before, operators didn't have a frame of reference so just went along with
it. Now they have more options and can more easily see the pain points in OpenStack and can decide to shift workload elsewhere. A single project can't make these sorts of overarching architectural decisions. The TC should do one of decide/help decide/facilitate deciding/delegate deciding. But someone needs to drive it, otherwise it gets dropped. That should be the TC IMO.
The TC can make people discussing, certainly. The TC can help arbitraring priorities, for sure. The TC can insufflate some guidance on archtectural decisions eventually. But the TC can't really identify the pain points for a specific project and allocate resources to work on those. TC folks aren't PMs or EMs.
The TC candidates are talking more and more about OpenStack being stable.
One development quote I like, "the code is done, not when there is nothing more to add, but nothing more to remove" speaks to me here... Do TC candidates think that should that be an architectural goal coming up soon? Figure out how to continue to do what OpenStack does, but do it simpler and/or with less code/services? That may require braking down some project walls. Is that a good thing to do?
I personnally expressed the idea to have the TC focusing on maturity gaps between projects. Having TC goals be aligned with efforts for filling those gaps is somehow something I look for. What you mention is slighly different, you'd propose a goal about reducing tech debt. I don't really see actionable items on this from a first sight, but it's worth to consider discussing it at the PTG.
-Sylvain
Thanks,
Kevin
*From:* Sylvain Bauza [sbauza@redhat.com] *Sent:* Thursday, February 21, 2019 9:28 AM *To:* Graham Hayes *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: [tc] Questions for TC Candidates
On Thu, Feb 21, 2019 at 6:14 PM Graham Hayes gr@ham.ie wrote:
On 20/02/2019 14:46, Chris Dent wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
The questions follow. Don't feel obliged to answer all of these. The point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
- How do you account for the low number of candidates? Do you consider this a problem? Why or why not?
I think we are reaching a more stable space, and the people who are developing the software are comfortable in the roles they are in.
As the demographic of our developers shifts east, our leadership is still very US / EU based, which may be why we are not getting the same amount of people growing into TC candidates.
- Compare and contrast the role of the TC now to 4 years ago. If you weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
4 years ago, was just before the big tent I think? Ironically, there was a lot of the same discussion - python3, new project requirements (at that point the incubation requirements), asyncio / eventlet.
The TC was also in the process of dealing with a By-Laws change, in this case getting the trademark program off the ground.
We were still struggling with the "what is OpenStack?" question.
Looking back on the mailing list archives is actually quite interesting and while the topics are the same, a lot of the answers have changed.
- What, to you, is the single most important thing the OpenStack community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
I think things like the review culture change have been good for this. The only other thing we can do is have more people reviewing, to make that first contact nice and quick, but E_NO_TIME or E_NO_HUMANS becomes the issue.
- If you had a magic wand and could inspire and make a single sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
1: Single agent on each compute node that allows for plugins to do all the work required. (Nova / Neutron / Vitrage / watcher / etc)
2: Remove RMQ where it makes sense - e.g. for nova-api -> nova-compute using something like HTTP(S) would make a lot of sense.
3: Unified Error codes, with a central registry, but at the very least each time we raise an error, and it gets returned a user can see where in the code base it failed. e.g. a header that has OS-ERROR-COMPUTE-3142, which means that someone can google for something more informative than the VM failed scheduling
4: OpenTracing support in all projects.
5: Possibly something with pub / sub where each project can listen for events and not create something like designate did using notifications.
That's the exact reason why I tried to avoid to answer about architectural changes I'd like to see it done. Because when I read the above lines, I'm far off any consensus on those. To answer 1. and 2. from my Nova developer's hat, I'd just say that we invented Cells v2 and Placement. To be clear, the redesign wasn't coming from any other sources but our users, complaining about scale. IMHO If we really want to see some comittee driving us about feature requests, this should be the UC and not the TC.
Whatever it is, at the end of the day, we're all paid by our sponsors. Meaning that any architectural redesign always hits the reality wall where you need to convince your respective Product Managers of the great benefit of the redesign. I'm maybe too pragmatic, but I remember so many discussions we had about redesigns that I now feel we just need hands, not ideas.
-Sylvain
- What can the TC do to make sure that the community (in its many
dimensions) is informed of and engaged in the discussions and decisions of the TC?
This is a difficult question, especially in a community where a lot of contributors are sponsored.
The most effective way would be for the TC to start directly telling projects what to do - but I feel like that would mean that everyone would be unhappy with us.
- How do you counter people who assert the TC is not relevant? (Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
Highlight the work done by the TC communicating with the board, guiding teams on what our vision is, and helping to pick goals. I think the goals are a great way, and we are starting to see the benifits as we continue with the practice.
For some people, we will always be surplus to requirements, and they just want to dig into bugs and features, and not worry about politics.
Thats fine - we just have to work with enough of the people on the teams to make sure that the project is heading in the correct direction, and as long as people can pick up what the priotities are from that process, I think we win.
That's probably more than enough. Thanks for your attention.
On 21/02/19 12:28 PM, Sylvain Bauza wrote:
> * If you had a magic wand and could inspire and make a single > sweeping architectural or software change across the services, > what would it be? For now, ignore legacy or upgrade concerns. > What role should the TC have in inspiring and driving such > changes? 1: Single agent on each compute node that allows for plugins to do all the work required. (Nova / Neutron / Vitrage / watcher / etc) 2: Remove RMQ where it makes sense - e.g. for nova-api -> nova-compute using something like HTTP(S) would make a lot of sense. 3: Unified Error codes, with a central registry, but at the very least each time we raise an error, and it gets returned a user can see where in the code base it failed. e.g. a header that has OS-ERROR-COMPUTE-3142, which means that someone can google for something more informative than the VM failed scheduling 4: OpenTracing support in all projects. 5: Possibly something with pub / sub where each project can listen for events and not create something like designate did using notifications.That's the exact reason why I tried to avoid to answer about architectural changes I'd like to see it done. Because when I read the above lines, I'm far off any consensus on those. To answer 1. and 2. from my Nova developer's hat, I'd just say that we invented Cells v2 and Placement. To be clear, the redesign wasn't coming from any other sources but our users, complaining about scale. IMHO If we really want to see some comittee driving us about feature requests, this should be the UC and not the TC.
Whatever it is, at the end of the day, we're all paid by our sponsors. Meaning that any architectural redesign always hits the reality wall where you need to convince your respective Product Managers of the great benefit of the redesign. I'm maybe too pragmatic, but I remember so many discussions we had about redesigns that I now feel we just need hands, not ideas.
C'mon, the question explicitly stipulated use of a magic wand, ignoring path dependence and throwing out backwards compat, but you're worried about the practicalities of convincing product managers??!?
We need to stop reflexively stifling these discussions. An 'open' community where nobody is allowed to so much as spitball ideas in case somebody disagrees with them is unworthy of the name.
- ZB
On Tue, 26 Feb 2019, Zane Bitter wrote:
We need to stop reflexively stifling these discussions. An 'open' community where nobody is allowed to so much as spitball ideas in case somebody disagrees with them is unworthy of the name.
Bless you.
Le mer. 27 févr. 2019 à 01:44, Zane Bitter zbitter@redhat.com a écrit :
On 21/02/19 12:28 PM, Sylvain Bauza wrote:
> * If you had a magic wand and could inspire and make a single > sweeping architectural or software change across the services, > what would it be? For now, ignore legacy or upgrade concerns. > What role should the TC have in inspiring and driving such > changes? 1: Single agent on each compute node that allows for plugins to do all the work required. (Nova / Neutron / Vitrage / watcher / etc) 2: Remove RMQ where it makes sense - e.g. for nova-api ->nova-compute
using something like HTTP(S) would make a lot of sense. 3: Unified Error codes, with a central registry, but at the veryleast
each time we raise an error, and it gets returned a user can see where in the code base it failed. e.g. a header that has OS-ERROR-COMPUTE-3142, which means that someone can google for something more informative than the VM failed scheduling 4: OpenTracing support in all projects. 5: Possibly something with pub / sub where each project can listenfor
events and not create something like designate did using notifications.That's the exact reason why I tried to avoid to answer about architectural changes I'd like to see it done. Because when I read the above lines, I'm far off any consensus on those. To answer 1. and 2. from my Nova developer's hat, I'd just say that we invented Cells v2 and Placement. To be clear, the redesign wasn't coming from any other sources but our users, complaining about scale. IMHO If we really want to see some comittee driving us about feature requests, this should be the UC and not the TC.
Whatever it is, at the end of the day, we're all paid by our sponsors. Meaning that any architectural redesign always hits the reality wall where you need to convince your respective Product Managers of the great benefit of the redesign. I'm maybe too pragmatic, but I remember so many discussions we had about redesigns that I now feel we just need hands, not ideas.
C'mon, the question explicitly stipulated use of a magic wand, ignoring path dependence and throwing out backwards compat, but you're worried about the practicalities of convincing product managers??!?
We need to stop reflexively stifling these discussions. An 'open' community where nobody is allowed to so much as spitball ideas in case somebody disagrees with them is unworthy of the name.
We are post the campaign period so I won't argue but see my other emails after this one, hopefully you will see that I'm not against discussing about architectural concerns, just not here and not by only the TC members.
Sylvain (stopping now the campaign)
- ZB
On 2019-02-21 17:10:03 +0000 (+0000), Graham Hayes wrote: [...]
The most effective way would be for the TC to start directly telling projects what to do - but I feel like that would mean that everyone would be unhappy with us.
[...]
To be clear, I can deal with people being mad at the TC if there's sufficient benefit to the community and users. But as the development of OpenStack is based on collaboration, dictating terms (rather than showing teams why some alternative path is a benefit for them) is likely to result in further shrinking the contributor base and/or forming forks under a less authoritarian regime. This is the real cost which must be weighed any time the TC considers using its power to force the hand of a team it governs.
Not sure my answer if fit one or some questions. But from my personal PoV, there is BIG gap between the expectation from contributors and the power of TC. As a cloud platform, though there are many components/services, their API should be highly consistent. The contracts between each other should be very stable. However, I don't think we did a great job around this. And also, we do have some cross project features are still weak, e.g. quota management, role management, etc.
And for the single most important thing I would like to do, is getting a most restrict API design cross project to make OpenStack like a whole building, but not building blocks, from user perspective.
On 21/02/19 3:46 AM, Chris Dent wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
The questions follow. Don't feel obliged to answer all of these. The point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
- How do you account for the low number of candidates? Do you
consider this a problem? Why or why not?
- Compare and contrast the role of the TC now to 4 years ago. If you
weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
- What, to you, is the single most important thing the OpenStack
community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
- If you had a magic wand and could inspire and make a single
sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
- What can the TC do to make sure that the community (in its many
dimensions) is informed of and engaged in the discussions and decisions of the TC?
- How do you counter people who assert the TC is not relevant?
(Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
That's probably more than enough. Thanks for your attention.
Chris Dent wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
Thanks Chris ! I'll try to make short answers, as the campaigning period unfortunately ended up overlapping with long-planned vacation :)
- How do you account for the low number of candidates? Do you
consider this a problem? Why or why not?
The TC activity traditionally took some significant time, so it was better suited to people who had the chance to spend 100% of their work time on OpenStack. As we mature, we have less and less people who can spend 100% of their time on OpenStack. Sometimes they have to share their time with other projects, or with their organization other priorities. It is therefore more difficult to find candidates with the available time. I don't think it is a problem in itself -- it is more a reflection of how our community changed and how much our systems remained the same.
- Compare and contrast the role of the TC now to 4 years ago. If you
weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
Having recently worked on the "role of the TC" document, for me it captures the role of the TC as it is. 4 years ago, we just finished evolving our systems to cope with massive growth. I think today we need to start evolving them again, with an eye toward long-term sustainability.
[...]
- What can the TC do to make sure that the community (in its many
dimensions) is informed of and engaged in the discussions and decisions of the TC?
Maybe we need to differentiate "communicating what we are doing as the TC" and communicating the global direction". OpenStack developers can ignore the former but should not ignore the latter. In the past we did converge both and hope everyone would read everything, and that was not very successful.
- How do you counter people who assert the TC is not relevant?
(Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
The TC is responsible for the whole of OpenStack, rather than the pieces (which are separately handled by project teams). We are in a good position to take a step back and make sure "OpenStack" looks good, beyond individual pieces. I personally think it is important, and it is why we are relevant.
On 20/02/19 9:46 AM, Chris Dent wrote:
It's the Campaigning slot of the TC election process, where members of the community (including the candidates) are encouraged to ask the candidates questions and witness some debate. I have some questions.
First off, I'd like to thank all the candidates for running and being willing to commit some of their time. I'd also like to that group as a whole for being large enough to force an election. A representative body that is not the result of an election would not be very representing nor have much of a mandate.
+1
The questions follow. Don't feel obliged to answer all of these. The point here is to inspire some conversation that flows to many places. I hope other people will ask in the areas I've chosen to skip. If you have a lot to say, it might make sense to create a different message for each response. Beware, you might be judged on your email etiquette and attention to good email technique!
- How do you account for the low number of candidates? Do you
consider this a problem? Why or why not?
In retrospect, it appears the tradition of waiting until the last possible minute had a lot to do with it. But I definitely had some other thoughts the day before nominations closed.
Looking at some of the folks who either decided not to run or who delayed their decision to run, I see a lot of standard, non-alarming reasons. Someone who is organising a wedding, someone who just changed jobs, someone who has a big internal project to work on at $DAYJOB, &c.
However, those things have always happened. The pool of candidates has shrunk to the point where folks who are borderline on having enough time to run are now the difference between having an election and having unfilled seats. That's in part due to the lower number of folks employed to work on OpenStack, and to the extra workload being shouldered by those left.
This is a problem for OpenStack, for at least the reason you mentioned above: TC members don't have much of a mandate if they didn't actually have an election.
Shrinking the TC is one option for dealing with this, and one we should probably at least investigate. But we could also work harder on developing future leaders of the project and getting them involved in the work of the TC (I guess the Stewardship WG used to be a more formal expression of this) before they eventually win seats on the TC - in much the same way that projects should hope to turn contributors into core reviewers.
- Compare and contrast the role of the TC now to 4 years ago. If you
weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
In some ways it's very different and in some ways depressingly familiar :)
Four years ago, the TC was focused on managing explosive growth of contributions (particularly in the form of new projects). The TC of that era did a lot of work putting governance into place that would make it possible to deal with the flood of new projects. That involved the TC removing itself from the technical review of new projects, in part because it had failed to give at least one project *any answer at all* on two successive graduation applications due to a lack of time on the part of TC members to investigate what it was for.
Fast-forwarding to the present, the flood has slowed to a trickle and the governance structures are mature, but the TC is still in many ways very focused on 'governance' to the exclusion of helping to get the technical community to work in concert.
- What, to you, is the single most important thing the OpenStack
community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
Since you're forcing me to pick only one thing, I am going to say 'education'.
The economics of participating in an open source project are just not intuitive to anyone who isn't steeped in this stuff, and most people are not. Most companies are vulnerable to acting especially clueless, because the folks making the decisions are not the same folks who would even have the chance to work in the community. Even most of the companies working on OpenStack in the early days were pretty bad at it, though almost all improved with practice.
Forking a rapidly-evolving project is one of the most expensive mistakes you'll ever make, and if people understood that I really don't believe they'd do it because the time-to-market gains or whatever are just not worth it, and soon evaporate when you're looking at the time-to-market for v2 of your now-unmaintainable product.
I feel like it's part of OpenStack's role in the world to explain the open source way - both the benefits it confers and the responsibilities it entails - to folks who are receptive but who were not closely associated enough with projects like Linux to have learned the ins and outs.
(If I get a second answer, it'd be the one I brought up at the Forum in Berlin: prioritising giving immediate feedback to any and all contributions from those folks. http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000052....)
- If you had a magic wand and could inspire and make a single
sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns.
Oh wow you went there.
If I had to pick one and I had a magic wand, I would replace the current backend infrastructure (comprising MariaDB, RabbitMQ, and at least theoretically etcd3) with a single component, with requirements such as the following:
- Efficiently scalable to some defined level - First-class support for messaging patterns (e.g. pub-sub) - Reliable to a passes-jepsen-or-better standard - Well-defined semantics for resolving conflicts/partitions/&c. - Support for large numbers of geographically distributed replicas (for edge use cases) - To the extent that any of this needs to be implemented in the OpenStack community, it only has to be implemented once
In response to the inevitable complaints:
- Yes, MySQL can be scaled using sharding. But we have 50+ projects and only 2 implementations of sharding. And to underscore just how difficult that is, both were in the same project. - Yes, RabbitMQ supports messaging patterns, but it's not something we've built our services around (e.g. Kubernetes is entirely event driven, but in OpenStack very little data escapes the compute node without querying for it) - in fact we mostly use it (inappropriately) for RPC. And it's a separate service from the database, so you can't e.g. have a transaction that includes sending a message. - RabbitMQ failed Jepsen spectacularly (https://aphyr.com/posts/315-call-me-maybe-rabbitmq) - if you can manage to keep it running at all. It's a constant source of complaints from OpenStack operators. Galera also failed BTW (https://aphyr.com/posts/327-call-me-maybe-mariadb-galera-cluster). - As an OpenStack developer, I can't even find any documentation to tell me what transaction isolation level I should expect from the DB. Not only is it not documented anywhere, but AFAICT until very recently we officially supported two *different* levels and at least tacitly encouraged operators to effectively choose for themselves which to use. How can I, as an OpenStack developer, even hope to write correct code under those circumstances? This is not theoretical; many times I've been in discussions that ended with us just guessing what the database would do and hoping for the best. I went searching fruitlessly for docs yet again because I literally spent half of Wednesday trying to implement something without having a race condition, and whether it was possible or not (spoiler: it's not) depended on the transaction isolation level. - Something something Edge. - See the first point above.
I'm not wedded to any particular choice, but I believe such things exist. I'm particularly intrigued by FoundationDB, although extremely wary of their non-open development model for the core.
Anyhow, with that in place I would wave my magic wand and build a messaging library and then a user-facing API that used it. Then I'd have everything in OpenStack work together as an event-driven system, with (select) events potentially routed out to userspace and back in again if the user so chose.
I don't think there's a path to this happening, but I think it's a very valuable exercise to imagine how things could be and think about what we're missing out on.
What role should the TC have in inspiring and driving such changes?
I don't think it should be the TC's job to necessarily come up with such ideas. Great ideas might come from anywhere. And I don't think it can be the TC's job to tell people what ideas to implement, because that's not how open source works.
Plagiarising myself (from https://review.openstack.org/#/c/622400/1/reference/role-of-the-tc.rst@73): "The ideas we need are out there in the community. The TC's job is to ensure that there's space for them to be heard, to amplify the best ones, to build consensus, to ensure that decisions get made when consensus is not possible (i.e. we don't default to paralysis when reasonable people disagree), and to be electorally accountable for getting it wrong."
- What can the TC do to make sure that the community (in its many
dimensions) is informed of and engaged in the discussions and decisions of the TC?
The TC is extremely open in how it conducts it's business. So the challenge, as always is in filtering the firehose.
I think the weekly TC update email from the chair is a big improvement, because it's significantly easier to engage with than trying to follow everything.
- How do you counter people who assert the TC is not relevant?
(Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
Trick question! I wouldn't try to counter them. If somebody believes that the TC is not relevant *to them* then they're almost certainly right. I would be interested in listening to what things _would_ be relevant to them and exploring whether those things should be done, whether they should be done by the TC, or whether the TC should delegate them to some other group. But let's remember that there will always be people who are just heads-down in their own project getting stuff done and never needing to worry about the TC, and that's fine. OpenStack can accommodate an ~unlimited number of those folks. But it can't prosper with only those folks. We need people who keep the lines of communication open between projects and shape the system as a whole. If the TC can't be relevant to that latter group, then we really have a problem.
cheers, Zane.
On Feb 25, 2019, at 8:01 PM, Zane Bitter zbitter@redhat.com wrote:
This is a problem for OpenStack, for at least the reason you mentioned above: TC members don't have much of a mandate if they didn't actually have an election.
That’s a good point: do you (all candidates; not just Zane) see the election as being a mandate for specific things? Candidates run on different platforms, expressing different desires for changes they would like to make. Do you see the result of a TC election as a mandate to go out and do those things, and not to do the things that the losing candidates espoused?
The counter, and extremely cynical, argument here is that people don’t really weigh the specific proposals of the individual candidates and choose those most in alignment with their feelings, but instead choose people who they either a) worked with at some point and didn’t find them to be a jerk, or b) have seen their name around for a while, and figure they must know what’s going on, or c) have the same employer, or d) some other non-issue-related reason. If this cynical point of view is closer to how you see reality, does that represent a mandate at all?
-- Ed Leafe
On Tue, Feb 26, 2019 at 10:23 AM Ed Leafe ed@leafe.com wrote:
On Feb 25, 2019, at 8:01 PM, Zane Bitter zbitter@redhat.com wrote:
This is a problem for OpenStack, for at least the reason you mentioned above: TC members don't have much of a mandate if they didn't actually have an election.
That’s a good point: do you (all candidates; not just Zane) see the election as being a mandate for specific things? Candidates run on different platforms, expressing different desires for changes they would like to make. Do you see the result of a TC election as a mandate to go out and do those things, and not to do the things that the losing candidates espoused?
Yes. I do think however that sometimes it can be really hard for us to do things as an individual if the rest of the committee disagrees
i.e.: i still think weekly meetings are useful, will get things done, will help us stay in sync and give an easily parse-able thing for the rest of the community to visit.. but, I haven't had success with that.
The counter, and extremely cynical, argument here is that people don’t really weigh the specific proposals of the individual candidates and choose those most in alignment with their feelings, but instead choose people who they either a) worked with at some point and didn’t find them to be a jerk, or b) have seen their name around for a while, and figure they must know what’s going on, or c) have the same employer, or d) some other non-issue-related reason. If this cynical point of view is closer to how you see reality, does that represent a mandate at all?
-- Ed Leafe
On 26/02/19 10:19 AM, Ed Leafe wrote:
On Feb 25, 2019, at 8:01 PM, Zane Bitter zbitter@redhat.com wrote:
This is a problem for OpenStack, for at least the reason you mentioned above: TC members don't have much of a mandate if they didn't actually have an election.
That’s a good point: do you (all candidates; not just Zane) see the election as being a mandate for specific things? Candidates run on different platforms, expressing different desires for changes they would like to make. Do you see the result of a TC election as a mandate to go out and do those things, and not to do the things that the losing candidates espoused?
I explained my theory about that in a previous ML post a few weeks ago:
http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001841.h...
In short, in order to co-ordinate across a group of people everyone in the group needs to have some reason to think the other people in the group are going to move in the same direction. Relaying the direction through an elected group helps to do that, because everyone believes that everyone else voted for that group - and in the aggregate they're correct.
I expect that effect would be significantly weakened (though not completely eliminated) if there was no election (which would mean that TC members effectively appointed themselves).
The counter, and extremely cynical, argument here is that people don’t really weigh the specific proposals of the individual candidates and choose those most in alignment with their feelings, but instead choose people who they either a) worked with at some point and didn’t find them to be a jerk, or b) have seen their name around for a while, and figure they must know what’s going on, or c) have the same employer, or d) some other non-issue-related reason. If this cynical point of view is closer to how you see reality, does that represent a mandate at all?
Yes, sadly I think it's likely that a & b at least play an outsize role (I _hope_ that c doesn't play a big role, and I do think that people consider actual issues or at least general philosophies to some extent). But interestingly it doesn't matter! The above effect turns out to still work even though it's just a convenient fiction, like money, or Belgium. It still works even though you all just read this message where I called it a fiction :)
cheers, Zane.
On Tue, Feb 26, 2019 at 4:25 PM Ed Leafe ed@leafe.com wrote:
On Feb 25, 2019, at 8:01 PM, Zane Bitter zbitter@redhat.com wrote:
This is a problem for OpenStack, for at least the reason you mentioned
above: TC members don't have much of a mandate if they didn't actually have an election.
That’s a good point: do you (all candidates; not just Zane) see the election as being a mandate for specific things? Candidates run on different platforms, expressing different desires for changes they would like to make. Do you see the result of a TC election as a mandate to go out and do those things, and not to do the things that the losing candidates espoused?
The counter, and extremely cynical, argument here is that people don’t really weigh the specific proposals of the individual candidates and choose those most in alignment with their feelings, but instead choose people who they either a) worked with at some point and didn’t find them to be a jerk, or b) have seen their name around for a while, and figure they must know what’s going on, or c) have the same employer, or d) some other non-issue-related reason. If this cynical point of view is closer to how you see reality, does that represent a mandate at all?
What you mention is the exact reason why I claimed in my candidacy ballot to have more folks for the election [1]. I don't remember since when we started to have campaigns for the TC election (at least 3 or 4 IIRC) but i hope it will help the election to not be just a popularity contest, but rather a way for voters to better know the individuals before giving them a ticket (and actually pounding this ticket using the Condorcet system if they really want to favor a candidate close to their own opinions).
If we were less or equal the number of TC seats, what would have been the representativity of such candidates who would have been elected by default ? Now, that's not because we have an election and that we have a campaign period that it will mean that those individuals will feel as 'members of parliament' (at least for me). One of the key values of OpenStack is Open Design on a consensus model. We all have great opinions and we care about our own opinions, but we also need a majority of contributors agreeing with.
-Sylvain
[1] https://git.openstack.org/cgit/openstack/election/plain/candidates/train/TC/... (last paragraph)
-- Ed Leafe
On Wed, Feb 20, 2019 at 10:47 PM Chris Dent cdent+os@anticdent.org wrote:
- How do you account for the low number of candidates? Do you consider this a problem? Why or why not?
I mostly concern about why previous TCs stop to consider join the election again? Multiple reasons indeed, but none of them concerns me. We need whoever qualified as a candidate to keep jump out and help. Still, this is a problem IMO, because we keep running our election between those we know how OpenStack community works here. And I sense more new joiner didn't actually know what exactly is TC or UC. There just to much history. more committee report outside of ML might help with that situation?
- Compare and contrast the role of the TC now to 4 years ago. If you weren't around 4 years ago, comment on the changes you've seen over the time you have been around. In either case: What do you think the TC role should be now?
I like some of the current tasks, including keep update our governance, but I would also like TC's role including reach out (Which I believe some TCs done a great job). We must keep reaching out with other parts of the community so we can actually make sure all flow is working. Like how can we get users/ops to provide experience, how can we get people from the global community to better understand what we needed now and how they can help, or how can we help with teams, UCs, etc. to fill up the gape between.
- What, to you, is the single most important thing the OpenStack community needs to do to ensure that packagers, deployers, and hobbyist users of OpenStack are willing to consistently upstream their fixes and have a positive experience when they do? What is the TC's role in helping make that "important thing" happen?
To guide them, and simplify their way through. Organization guideline [1] is definitely a star, if we actually plan to promote it.
As for simplifying the way, we have too many ways to achieve too many things, but lack of plans to integrate or even lack documentation to show them why we got so many things. The most famous words we using is `that's because of some history`, but when we fail on provides a guideline or future plan, history will keep going. Which is up to TC to guiding teams to achieve that goal.
- If you had a magic wand and could inspire and make a single sweeping architectural or software change across the services, what would it be? For now, ignore legacy or upgrade concerns. What role should the TC have in inspiring and driving such changes?
remove rabbitmq dependancy, centralize API implementation (import oslo.api), integrate TCs with UCs into a single commitee(consider that as community architectural:) ).
BTW, after I'm done with it, I might also think about selling it. :)
- What can the TC do to make sure that the community (in its many dimensions) is informed of and engaged in the discussions and decisions of the TC?
The best way is to post on the place people actually read, also on the place people might be interested to learn more about us. It requires collaboration all the way from TC to end users (like user groups, openInfra events, or enterprises). I mean TC+UC kind of collaboration.
- How do you counter people who assert the TC is not relevant? (Presumably you think it is, otherwise you would not have run. If you don't, why did you run?)
That's why we need to keep document things down and keep making OpenStack as success as it can be. So people get to know better if it's relevant to them or not. Try to figure out if we didn't provide enough information or action to let people understand the scope of TC's responsibility.
That's probably more than enough. Thanks for your attention.
-- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
[1] https://docs.openstack.org/contributors/organizations.
-- May The Force of OpenStack Be With You, Rico Lin irc: ricolin
participants (16)
-
Alexandra Settle
-
Chris Dent
-
Ed Leafe
-
Feilong Wang
-
Fox, Kevin M
-
Graham Hayes
-
Jeremy Stanley
-
Jim Rollenhagen
-
Lance Bragstad
-
Matt Riedemann
-
Mohammed Naser
-
Rico Lin
-
Sean Mooney
-
Sylvain Bauza
-
Thierry Carrez
-
Zane Bitter