[openstack-dev] [tc] [all] [glance] On operating a high throughput or otherwise team
Nikhil Komawar
nik.komawar at gmail.com
Tue May 17 04:36:21 UTC 2016
Thanks for your reply. No offense intended (on the earlier one). I
really want us to make sure all of us are on the same page on this thread.
On 5/16/16 11:32 PM, John Griffith wrote:
>
>
> On Mon, May 16, 2016 at 7:43 PM, Nikhil Komawar <nik.komawar at gmail.com
> <mailto:nik.komawar at gmail.com>> wrote:
>
>
> First of all, are you serious? This is not the right email thread for
> such issue complaints. This thread is about communication and not just
> for Glance but for [tc] and [all]!
>
> Secondly, you have not mentioned _anything_ about the upgrade
> numbers/releases. You may want to notice the *major* API upgrade
> to the
> glance-api server you are talking to (hint is_public vs. visibility),
> possibly the reason for all your issues.
>
> Thirdly, when the client is upgraded please check the default
> version it
> is going to use, it's in the release notes, etc.
>
> We are more than happy to help resolve the issues if done rightly.
> Dumping stuff on the ML after a bad customer call is a bit... sad
> (if I
> were to put it mildly). You can talk to us separately and we will be
> happy to point you to the operators who are running large scale
> OpenStack installation & actually happy with OpenStack services that
> include Glance.
>
> And, I think you just proved the point I'm making, is ML the right
> place
> to share ideas sanely? Do we have etiquette that people actually
> follow
> making sure you stay on topic and move forward, or rather diverge and
> create even more problems?
>
>
> On 5/16/16 9:06 PM, John Griffith wrote:
> >
> >
> > On Mon, May 16, 2016 at 10:10 AM, Flavio Percoco
> <flavio at redhat.com <mailto:flavio at redhat.com>
> > <mailto:flavio at redhat.com <mailto:flavio at redhat.com>>> wrote:
> >
> > On 16/05/16 00:23 -0700, Clint Byrum wrote:
> >
> >
> > Excerpts from Nikhil Komawar's message of 2016-05-14
> 17:42:16
> > -0400:
> >
> >
> > Hi all,
> >
> >
> >
> >
> >
> > Lately I have been involved in discussions that have
> > resulted in giving
> >
> > a wrong idea to the approach I take in operating the
> > (Glance) team(s).
> >
> > While my approach is consistency, coherency and
> agility in
> > getting
> >
> > things done (especially including the short, mid as well
> > as long term
> >
> > plans), it appears that it wasn't something evident.
> So, I
> > have decided
> >
> > to write this email so that I can collectively gather
> > feedback and share
> >
> > my thoughts on the right(eous) approach.
> >
> >
> >
> >
> >
> >
> > I find it rather odd that you or anyone believes there is a
> > "right"
> >
> > approach that would work for over 1500 active developers
> and 200+
> >
> > companies.
> >
> >
> >
> > We can definitely improve upon the aspects of what we
> have now, by
> >
> > incremental change or revolution. But I doubt we'll ever
> have a
> >
> > community that is "right" for everyone.
> >
> >
> >
> >
> >
> >
> > Right! To that let's also add we have a bunch of smaller
> > communities within our
> >
> > OpenStack big tent. Each one of these teams might requires a
> > different approach.
> >
> >
> >
> > [snip]
> >
> >
> >
> >
> >
> >
> > We are developing something that is usable,
> operationally
> > friendly and
> >
> > that it's easier to contribute & maintain but, many
> strong
> > influencers
> >
> > are missing on the most important need for OpenStack --
> > efficient way of
> >
> > communication. I think we have the tools and right
> > approach on paper and
> >
> > we've mandated it in the charter too, but that's not
> > enough to operate
> >
> > things. Also, many people like to work on the assumption
> > that all the
> >
> > tools of communication are equivalent or useful and
> there
> > are no
> >
> > side-effects of using them ever. I strongly disagree.
> > Please find the
> >
> > reason below:
> >
> >
> >
> >
> >
> >
> > I'd be interested to see evidence of anyone believing
> > something close
> >
> > to that, much less "many people".
> >
> >
> >
> > I do believe people don't take into account everyone's
> > perspective and
> >
> > communication style when choosing how to communicate. But we
> > can't really
> >
> > know all of the ways anything we do in a distributed system
> > affects all
> >
> > of the parts. We can reason about it, and I think you've
> done
> > a fine job
> >
> > of reasoning through some of the points. But you can't know,
> > nor can I,
> >
> > and I don't think anyone is laboring under the illusion that
> > they can
> >
> > know this.
> >
> >
> >
> >
> > This is a good point and I believe it may explode into
> several smaller
> >
> > discussions. I'll try to light the bomb:
> >
> >
> >
> > - We used to be a community that assumed good faith about
> people,
> > proposals,
> >
> > etc. Have I been genuine enough to believe this is still
> true? I
> > certainly
> >
> > work under this *assumption* unless I things stink really
> bad and
> > even then,
> >
> > I'd try to work around the issue without unecessary
> finger-pointing.
> >
> >
> >
> > - We've always been a multi-cultural community company-wise,
> tz-wise,
> >
> > culturally-wise, language-wise, etc. These is super hard to
> > coordinate and
> >
> > finding a right solution for everyone is even harder. For
> > example, depending
> >
> > the communication medium you might have a bad/good impact on
> > non-native
> >
> > English speakers. Spending enough time understanding people's
> > perspective is
> >
> > critical to avoid the frustration of non-native English
> speakers.
> > Asking dumb
> >
> > questions that translate to "I don't get your English" when
> > things are utterly
> >
> > clear doesn't help, really.
> >
> >
> >
> > - Is picking the communication tool based on people's
> preferences
> > rather than
> >
> > based on the technical issue they are meant to solve the right
> > thing to do?
> >
> > I'm sorry if this is missing part of your point but I believe
> > each one of the
> >
> > tools we have are meant to ease specific communication
> issues for
> > specific
> >
> > cases. That is to say, I agree not all mediums are
> equivalent but
> > I do think
> >
> > there must be a preferred medium for "whenever you don't know
> > where to send
> >
> > $X". To me, that's the ML.
> >
> >
> >
> >
> >
> > [snip]
> >
> >
> >
> >
> > [1] https://en.wikipedia.org/wiki/Conway's_law
> <https://en.wikipedia.org/wiki/Conway%27s_law>
> > <https://en.wikipedia.org/wiki/Conway%27s_law>
> >
> >
> >
> >
> >
> >
> > * So, what can be the blocker?
> >
> >
> >
> > Nothing, but working with these assumptions is
> really the
> > blocker. That
> >
> > is exactly why many people in their feedback say we
> have a
> > "people
> >
> > problem" in OpenStack. But it's not really the people
> > problem, it is the
> >
> > assumption problem.
> >
> >
> >
> > Assumptions are very very bad:
> >
> >
> >
> > With 'n' problems in a domain and 'm' people working on
> > all those
> >
> > problems, individually, we have the assumption
> problem of
> > the order of
> >
> > O((m*e)^n) where you can think of 'e' as the convergence
> > factor.
> >
> > Convergence factor being the ability of a group to
> come to
> > an agreement
> >
> > of the order of 'agree to agree', 'agree to
> disagree' (add
> > percentages
> >
> > to each for more granularity). There is also another
> > assumption (for the
> >
> > convergence factor) that everyone wants to work in the
> > best interest of
> >
> > solving the problems in that domain.
> >
> >
> >
> >
> >
> >
> >
> >
> > rAmen brother. We can't assume to know the motivations of
> > anyone, though
> >
> > we can at least decide how much to trust what people say. So
> > if they say
> >
> > that they're interested in solving the problems in a
> domain, I
> > certainly
> >
> > will give them space to prove that right or wrong.
> >
> >
> >
> >
> > Ok, let me see if I can put this in a way it doesn't sound bad.
> > Bear with me.
> >
> >
> >
> > Sometimes, when it comes to keeping communications
> healthy/going in
> >
> > multi-cultural communities, it's better to have a bit of
> > "assumption". Depending
> >
> > on the problem's space, topic, etc, it might be healthier to
> > assume some things
> >
> > on specific points than keep poking with the same questions
> > (structured
> >
> > differently) which just end up frustrating folks. I guess my
> point
> > is, there's a
> >
> > time and space for every discussion/question. Unfortunately, I
> > believe we don't
> >
> > spend enough time understanding this and we as well sometimes
> > disagree on what's
> >
> > the right time/space to discuss things.
> >
> >
> >
> > Hope the above makes (some) sense. As a general thing, I
> prefer to
> > assume good
> >
> > faith, it truly makes my life simpler.
> >
> >
> >
> > [snip]
> >
> >
> >
> >
> > * Also, one very important thing that I keep hearing: "I
> > do not like
> >
> > that" without any other information, as an argument to
> > disregard
> >
> > technical proposals. I think it is very disruptive and
> > irrational way to
> >
> > express arguments. We are not buying flowers in
> OpenStack,
> > we need to
> >
> > keep rationality in check when we express our
> opinions. It
> > reduces
> >
> > convergence factor and increases dubiety among the
> > developers &
> >
> > reviewers. Then we have a ecosystem where people do not
> > understand why
> >
> > we do things the way we do it. We should not stop
> > businesses just
> >
> > because someone doesn't like something, please no.
> Lack of
> > rationale can
> >
> > actually do that.
> >
> >
> >
> >
> >
> >
> >
> >
> > +1! Qualify or quantify your objections, but please
> don't just
> > give us
> >
> > your unfiltered opinions.
> >
> >
> >
> >
> > ++
> >
> >
> >
> > [snip]
> >
> >
> >
> >
> > As a counterpoint, I think we mostly need more understanding
> > of our
> >
> > distributed nature, and to let go of the idea that we can
> > control any
> >
> > of it. To anyone involved I say: Wield your influence, and
> > measure your
> >
> > success, but don't expect 1500 people to do what you
> tell them
> > to do,
> >
> > because they might just have 1500 different ideas of
> what you
> > actually
> >
> > meant.
> >
> >
> >
> >
> > This! This! and again, THIS!
> >
> >
> >
> > Thank y'all,
> >
> > Flavio
> >
> >
> >
> > --
> >
> > @flaper87
> >
> > Flavio Percoco
> >
> >
> >
> __________________________________________________________________________
> >
> > OpenStack Development Mailing List (not for usage questions)
> >
> > Unsubscribe:
> >
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> >
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> >
> >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> > Hey,
> >
> > Maybe not related, but maybe it is. After spending the past
> couple of
> > hours trying to help a customer with a Glance issue I'm a bit...
> well
> > annoyed with Glance. I'd like to chime in on this thread. I'm
> > honesty not entirely sure what the goal of the thread is, but
> honestly
> > there's something rather important to me that I don't really seem to
> > see being called out.
> >
> > Is there any way we could stop breaking the API and it's behaviors?
> > Is there any way we can fix some of the issues with respect to how
> > things work when folks configure multiple Glance repos?
> >
> > Couple of examples:
> > 1. switching from "is_public=true" to "visibility=public"
> >
> >
> > Ok, cool, I'm sure there's great reasons, but it really
> sucks when
> > folks update their client and now none of their automation works
> > any longer
> >
> >
> > 2. making virtual_size R/O
> >
> >
> > So for some time this was a property that folks could use to set
> > the size of an image needed for things like volume creation,
> > cloning etc. At some point though it was decided "this
> should be
> > read only", ok... well again all sorts of code is now broken,
> > including code in Cinder. It also seems there's no way to set
> > it, so it's always there and just Null. It looked like I
> would be
> > able to set it during image-create maybe... but then I hit
> number 3.
> >
> >
> > 3. broken parsing for size and virtual_size
> >
> > I just started looking at this one and I'm not sure what
> happened
> > here yet, but it seems that these inputs aren't being parsed any
> > more and are now raising an exception due to trying to stuff a
> > string into an int field in the json schema.
> >
> >
> > So I think if the project wants to move faster that's great, but
> > please is theres any chance to value backwards compatibility just a
> > bit more? I'm sure I'm going to get flamed for this email, and the
> > likely response will be "you're doing it wrong". I guess if I'm the
> > only one that has these sorts of issues then alright, I deserve the
> > flames, and maybe somebody will enlighten me on the proper ways of
> > using Glance so I can be happier and more in tune with my Universe.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
>
>
> --
>
> Thanks,
> Nikhil
>
>
>
> >
> First of all, are you serious?
>
> Yes
>
> >
> This is not the right email thread for
> >
> such issue complaints. This thread is about communication and not just
> >
> for Glance but for [tc] and [all]!
>
> Maybe you're right, but I think there is some relevance here.
> Communication is important, doing things "open" is important.
> Developing projects in a vacuum doesn't end well. You could write the
> coolest, trickiest, most novel piece of software in the world, but if
> it sucks for end-users to try and consume it, or changes in weird
> little ways every 6 months, it's just a waste.
Sure and I work on OpenStack because we are an open community. But
open-ness should not come at the cost of sanity of conversations. We are
going into the subjective field of what vacuum means and what level of
collaboration is essential.
This thread exists not as a result of one or two experiences, I've
observed changes over more than 2 years and have written this down,
probably in a phase when I was triggered by some events and behavior
that I did not expect from the revered members of the community.
I think you hit a point where your experience with the upgrade was
painful so, I will accept the comment on the weird little ways. But
software is dynamic, it changes with every commit and that's the nature
of the reality. Can we make it better? -- we're working on a few things
that will make sure OpenStack fits the requirements of those concerned.
For example, if you use Glance as just an internal service and ask
operators who want to scale it massively, you will get a very
contradictory response to what you just described where the focus is
more on scale. So, in those cases we need to make major upgrades to the
API that will involve changes. I am not suggesting that we will do this
in OpenStack -- it is my thought on how different priorities in
OpenStack result into different focus.
>
> So what does that have to do with your thread and communication? The
> fact is, that the size and number of key contributors involved in
> OpenStack and the tight coupling between the projects prohibits using
> traditional meetings as the primary form of communication or even
> IRC. I used to be able to just troll #openstack-dev, #openstack and
> #openstack-meeting all day and keep on top of things. That doesn't
> work any more.
That sounds fair. And, if you read my thread carefully and completely
(not in pieces) you will realize that I mention something about intent
not being clear in the ML emails. This is a thread that talks about
optimal communication -- you need that sometimes and you don't most
times. I do talk about being judicious use of ML and not asking about
abandoning it.
Being judicious is very very important otherwise, open collaboration
becomes more of a disruption than any help. Collaboration is about
integration of ideas, where divergence in that thought stream results
into disruption that affects not just one but many (sub-)groups.
Your one angry reply can result into a train wreckage for other set of
people. Being constructive and rational in the approach is exactly what
I have talked about in the first email as well.
>
> Don't get me wrong, any project should feel free to do whatever they
> want to work as a sub-group together SO LONG as they do everything
> possible to include any and all who might want to participate. If
> they openly enable and welcome participation in whatever forum they
> choose... then hey, more power to them. In addition however, I think
> they still have a responsibility to collaborate and share with the
> broader community. W
> hen it comes to the broader eco-system there's no way for people to
> keep apprised or provide feedback to all of the various projects other
> than the ML. TimeZones, conflicting meetings and the amount of change
> just don't allow many alternatives.
I agree to all of that, and again being judicious is important.
>
> You mentioned communicating through code, I like that; but that's
> difficult too. The rate of change in OpenStack as a whole prohibits
> most of us from really being able to keep up with everything that's
> going on. Then you end up just moving some of the problems you have
> with the ML into the review system which frankly can get bogged down
> enough on it's own already just based on technical disagreements.
I think we need to be careful when and where we raise concerns. It is
important to be mindful about the time and completeness of the comments
if you are raising serious blockers. Also, we need to make sure to be on
point and not generate speculation (I've been on other side and have
realized the error).
For example, what's a good time to raise a concern on ML (when you know
distractions are real)? 1 month, 2 months, 6 months, 1, 2, 3 years?
Let's all be reasonable on the ask.
>
> So yes, I pointed out some examples of things that annoyed me today,
> perhaps that was inappropriate. In my opinion their relevant points,
> but I won't light the fire again. So, for the distraction I may have
> caused you, I apologize, but the take away for me (without writing a
> novel) is if you want to deliver a consistent, reliable and quality
> product you have to have open and accessible communication not only on
> a sub-team but across the entire community where everyone is impacted
> by your decisions. I do not know of a better, more inclusive way of
> doing that than public ML's. Sure they can get derailed by folks like
> me (it happens), but that's a trade off I can live with. Besides, I
> can always hit the gmail "mute" button.
Thanks for bringing us back to the point.
Like I said, we are more than happy to help. Let's make sure we discuss
with the right topic and involve the right people so that we can
actually work on alleviating those issues.
Precision in the discussion is equally (or probably more) important as
much as the diversity in the feedback.
How can we all as a community come up with focused and timely feedback?
I've only attempted to construct the foundation for that discussion --
whether people actually think that's relevant or not is their own choice.
A lot of people have replied that there are similar issues but we are
not doing anything about it. So, are we okay conveniently ignoring a
real problem?
>
> Thanks,
> John
>
>
>
--
Thanks,
Nikhil
More information about the OpenStack-dev
mailing list