Open Stack

On 09/07/2011 04:13 AM, Sandy Walsh wrote:
> Thanks for the reply Monty.

My pleasure! Thanks for both your interest and your energy on the subject!

For the record ... bugs can be filed against any element of the CI
system at:

https://bugs.launchpad.net/openstack-ci

> The major argument I've heard to date about using something other than Gerrit is the effort gone into tying into the CI system. I buy those arguments and support not reinventing the wheel. Roundabout seemed like the logical point of integration for hubcap, but if there are other efforts that get us further for less work, I'm completely open to that.
>
> All your issues about single state and status pages are what Hubcap intended to resolve.

I hear that - one of the other things we were trying not to do is write
our own system. We are in the business of writing cloud software, not CI
systems - and I would strongly prefer customizing/fixing an existing
system and submitting patches upstream for ongoing maintenance than
writing our own software from scratch and then needing to maintain it
moving forward.

Because of the lag between us fixing something and it landing upstream
(and also because of the lag between us fixing and filling out the
initial Rackspace paperwork to be allowed to submit upstream) we are
currently running a gerrit that has two local patches applied to it.
This does mean that we are set up to run locally modified gerrit -
although we'd like to get out of the business of doing this eventually.

That being said - the dev effort that is going in to alternate systems
could, you know, go in to the gerrit that we're using. I'd be thrilled
to have folks pitch in and help on that front. :)

> The other arguments about parsing comments from github pull requests are less convincing to me than:

(Thank you, btw, in advance, for a specific list. It's honestly very
helpful. When I respond with things like "I have not experienced this" -
I'm not being snarky - I honestly need more information so that I can
attempt to solve the problem)

> 1. the error-prone Change-ID mechanism that Gerrit uses

This has not been a problem for me so far. But let me explain the
motivation behind this.

The git workflow that we're modeling is very similar to this:

http://sandofsky.com/blog/git-workflow.html

Which asserts (I believe quite correctly) that it is preferable to have
each new change that lands in trunk land as a single and complete patch.
I'm not going to repeat the arguments made in the post above, they're
really excellent.

An argument could be made that we should just trust the devs to manage
this themselves, and in my core I agree with that. The reality, though,
is that I get feedback from folks like this:

"I've hired a bunch of new graduates recently, and they always seem to
find the nearest length of rope to hang themselves with."

Which puts us in a little bit of a tricky situation... how do we empower
our power devs while still allowing our recent grads to not screw
everything up? (it's a fun juggling match, I assure you)

In any case, to accomplish the clean trunk history, one must rebase
changes on top of origin/master before submitting them. Then, if you get
review feedback and fix your branch, you need to amend the commit or you
need to rebase/squash on master again. Once you've done that, you have a
new commit - so there is really no way that gerrit can know that your
new change is just a new version of the old change without the Change-Id.

Part of this also comes from a semantic difference in how github and
gerrit view the world. On github, you develop on your personal fork, and
then you submit one of the branches in your fork to be pulled - so the
unit of review is the branch- meaning that amended changes can be
tracked. The thing you lose, though, is that you have to switch from
your dev environment to your web browser to submit a change for review.

With gerrit, you are submitting changes for review (kind of closer to
the Linus version of git where people are submitting patches) and
branches exist only for your personal organizational needs. Doing this
means that you can submit simply by doing "git review" OR "git push
gerrit HEAD:refs/for/master" if you don't like the alias/script.

In any case, there's one thing you need to learn about Change-Id
comments, which is that when you rebase/squash or ammend a commit that
you need to keep them and they need to be at the end of the message.
Once you learn that - it's really not much of a problem.

> 2. the gerrit UI that opens 30 tabs when you want to do a review

This is a known bug and has a patch upstream from Nokia/Qt:

http://code.google.com/p/gerrit/issues/detail?id=938
https://bugreports.qt.nokia.com/browse/QTQAINFRA-174

It is on or todo list to either upgrade when this lands in trunk, or
investigate backporting the patch if we get the bandwidth.

> 3. the gerrit UI that loses your review data randomly

I have not experienced this. Can you elaborate?

> 4. the environment variables you have to remember to set in each project

I set no environment variables in each project. In fact, I don't set
anything up per-project. I'm very curious about what you're needing to
do here.

Unless... it's possible we have something unclear in the docs.

The docs for the initial setup commands list things like:

USERNAME=foo

which we did just to make the docs more readable. (If you're setting up
a git remote, and the username on the remote host is different than the
username on your local machine, you're going to need to add a username
into the ssh connection string. Thus, $USERNAME.)

If this is what you're talking about - I'll certainly see if we can't
clarify the docs.

> 5. the 'git review' mechanism that prevents me from using 'git push' as expected.

You can use git push. What is it you are expecting to be able to do
here? What can you not do?

> 6. the git dances that have to be performed when something goes wrong

This is true of any git workflow with a large enough set of people that
is attempting to maintain a trunk with clean history. git is an
amazingly powerful tool and anybody using the system is going to need to
understand git, git history and git rebasing in order to do anything
tricky. The same would ultimately be true of github, the problem would
just show up later in the cycle.

> 7. the gerrit remotes that have to be set up that break your mental model of how git remotes work

Again, I'm going to ask you to be explicit on explaining your mental
model of how git remotes work.

> 8. complete re-votes required after a re-review (unless that was intentional?)

That is intentional. After new changes are pushed, the code is, in fact,
new, and needs to be reviewed. However, the old reviews are still there
for reference.

> ... and that's only from my first few days using Gerrit. 

I should point out that there are several things we have in place to
simplify the simple case. It's possible you are the type of person who
would like to know more. It is entirely possible to use gerrit without
the git review alias. It's all based on git push, actually ... you just
have a to learn a couple of things. Would you be interested in looking
at the man behind the curtain? It's entirely possible we've made
incorrect assumptions about which bits should be hidden, and I'd love
feedback on that.

> Moreso, as was stated before, why are we even using github in this case? Why not just stand up a git server? 
> Right now it feels like bubble-gum and binder twine.

github is simply a mirror of the canonical git repository, which is
maintained by gerrit. We mirror the code to github to make it easy for
people to discover and fork. We had several discussions with the PPB
about not mirroring to github, but it was decided that it was nicer for
people if we mirrored.

You can, however, COMPLETELY use this system without ever a single time
touching github.

> We're talking simple string parsing here. The last keyword from a user is that users vote. Multiple pull requests would be equally easy to support with a !new_vote command (or some such thing).
> 
> I'm sure I'm missing some critical piece of data here. 

I don't want to extend any false senses here. We are no longer examining
roundabout or hubcap as possibilities. Transitioning workflow tooling
for a project like OpenStack is a huge and caustic endeavor. When the
PPB voted that projects would all use consistent tooling and that the
tooling would be git/gerrit for code hosting/review, the understanding
was that we were uninterested in re-addressing the question every time a
dev had a new idea of how we could do things. As you said in another
email, currently the code is just a hack (it's a nice hack, but it's
just a start) in order to even begin to conceive of using it in
production for all of openstack, it would have to be finished. Then we'd
have to work through how we'd organize and run things, how we'd
transition. We'd have to write docs. We'd have to do testing. We'd have
to work on the surrounding tooling.

Then, we'd have to go get buy in from the bazillion different
stakeholders in this project. We did, in fact, just recently, get
everyone's buy in on gerrit - including some folks who prefer
bzr/launchpad. (In fact, I was one of the people, if you'll remember,
who preferred bzr/launchpad. Life is about compromise)

So, with all due respect - work on hubcap if you like it as a project
and feel like hacking on it. I'm sure it will provide joy for some
projects out there just as roundabout has - but please understand that
OpenStack is not considering using it.

We _are_, on the other hand, quite strongly interested in bugs and/or
patches from folks addressing the current system.

Thanks!
Monty

> ________________________________________
> From: Monty Taylor [mordred at inaugust.com]
> Sent: Tuesday, September 06, 2011 11:31 PM
> To: Sandy Walsh
> Cc: Stefano Maffulli; openstack at lists.launchpad.net
> Subject: Re: [Openstack] A possible alternative to Gerrit ...
> 
> Hi everybody!
> 
> I understand some of you are not comfortable with Gerrit and the
> workflow I and others are working to implement. While this may be a
> problem for some, it was never our intention to make life hard for
> anybody here. Let me try to explain why and how we got to this decision
> and make a proposal on how to move forward.
> 
> OpenStack is already a large project with tons of contributors, and it
> aims to continue to be larger over time. This is one of the reasons for
> all of the developer automation and controls that we put in back at the
> very beginning - what works well for a team of 5 doesn't necessarily
> work well for teams of 100+, especially when many of those 100+ may be
> corporate contributors only hacking on the code because they are being
> paid to. We have many corporate partners in this project, and over time
> the number of devs that fit that profile is very likely going to increase.
> 
> The CI team led the analysis of a move to git and github and found
> several deficiences in github pull requests, mainly in the area of
> failure conditions and scale. They work GREAT for handling the simple
> case, when they're just used as a basis for code reviews, or when every
> thing works perfectly the first time. One major limitation, for example,
> is that things get quickly complex when code needs to get re-reviewed as
> part of normal process. Another is that github does not provide a
> project-management oriented overview of pull request states.
> 
> With pull requests, it is impossible to see the current state of a
> review, because there IS no current state of a review other than open or
> closed. As far as we can tell, the general practice with pull requests
> in this regard is to close the pull request on rejection and open a new
> one when the review has been addressed and a new review is desired. The
> problem with that is the loss of historical context for reviews. When
> you have tons of reviewers and reviews, knowing that someone is just
> fixing a review comment and that the entire change doesn't need to be
> re-reviewed from scratch is helpful.  This feature is very important for
> the Project Tech Leads.
> 
> We asked for months and months for conversations with github
> about adding an overall status field and were told that the folks at
> github were not interested in doing this.
> 
> A meta system can be hacked in to overload pull request comments in a
> way that systems like roundabout can use, but this ends up again being
> excellent for the simple case of approval or denial, but becomes baroque
> when a pull request is sent back for review several times.
> 
> We couldn't find a convenient way to overcome this limitation without
> adding complicated steps and extra knowledge for developers.
> 
> We looked at roundabout- quite strongly, in fact. It was, indeed, our
> first choice when considering how to integrate github with our developer
> systems. It's a great tool and I've recommended it to other people given
> their workflow needs. However, in addition to the fact that it's based
> on magical text in github pull request comments as I mentioned above, it
> also does not have the level of jenkins integration that gerrit does.
> The Gerrit Jenkins plugin (which is developed and maintained by the fine
> folks at Nokia, but for which we've been developing patches to send
> upstream) has deep hooks in to Jenkins. It can have jobs respond to the
> gerrit event stream (no polling repos - stuff happens immediately) and
> more importantly can converge/combine any jobs listening to the same
> event as needing to all succeed. The sucess or failure of any of the
> jobs is then reported back to the review in a sensible way - and the
> workflow state transitions are quite clear. The two work in concert to
> provide all of the change landing ability and flexibility we need moving
> forward. Even better - it's an active project with an active upstream (a
> combination of Google, Nokia and Qt no less) so we don't have to be in
> the business of writing a custom CI integration piece.
> 
> Most importantly - we have had this conversation multiple times. We've
> been discussing this publicly for months. We have addressed concerns
> and enumerated the reasons why gerrit is technically superior given our
> requirements.
> 
> We all know that nothing is perfect and we're all about change:
> OpenStack is a big project and it's young. There will always be changes.
> We've already changed our VCS after only one year. Anybody is of course
> free to make his/her pitch for a different tool. Remember though that
> every change doesn't come cheap and the cost needs to be justified in
> front of tech leads and the hundreds of developers busy on the project.
> 
> With this many devs, there will NEVER (and I cannot stress that word
> never enough is a textual email) be full agreement on developer tooling.
> However, what we can do is take in the input of everyone's needs and do
> our best to design a system that meets those needs in a way that serves
> the project as a whole. That this inevitably means that the needs or
> preferences of individual developers may not be stroked is merely a
> given with a project of this size.
> 
> My suggestion is that folks send feedback on specific issues you have
> with the current git/gerrit/jenkins setup and we'll be happy to do what
> we can to address them.
> 
> Thanks!
> Monty
> 
> On 09/05/2011 10:14 AM, Sandy Walsh wrote:
>> Absolutely. It's a holiday here (I'm just checking in periodically).
>>
>> Enjoy your day y'all!
>>
>> -S
>>
>> ------------------------------------------------------------------------
>> *From:* openstack-bounces+sandy.walsh=rackspace.com at lists.launchpad.net
>> [openstack-bounces+sandy.walsh=rackspace.com at lists.launchpad.net] on
>> behalf of Stefano Maffulli [smaffulli at gmail.com]
>> *Sent:* Monday, September 05, 2011 12:35 PM
>> *To:* openstack at lists.launchpad.net
>> *Subject:* Re: [Openstack] A possible alternative to Gerrit ...
>>
>> 2011/9/5 Sandy Walsh <sandy.walsh at rackspace.com
>> <mailto:sandy.walsh at rackspace.com>>
>>
>>     That said, whether we use roundabout or use the code that has
>>     already been created for the gerrit/jenkins integration is perhaps
>>     worthy of a discussion?
>>
>>
>> I believe it is worth a discussion. Since today is a holiday in the US
>> and many developers are offline I propose we pause this discussion for
>> the rest of the day. I'll talk to Jay, Monty and Jim tomorrow and will
>> come back with a plan to address the concerns raised on the list. Would
>> that work for you?
>>
>> /stef
>> This email may include confidential information. If you received it in
>> error, please delete it.
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack at lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> This email may include confidential information. If you received it in error, please delete it.
> 
>