Open Stack

On Mon, 27 Aug 2018 10:31:50 -0500, Matt Riedemann wrote:
> On 8/24/2018 7:36 AM, Chris Dent wrote:
>>
>> Over the past few days a few of us have been experimenting with
>> extracting placement to its own repo, as has been discussed at
>> length on this list, and in some etherpads:
>>
>> https://etherpad.openstack.org/p/placement-extract-stein
>> https://etherpad.openstack.org/p/placement-extraction-file-notes
>>
>> As part of that, I've been doing some exploration to tease out the
>> issues we're going to hit as we do it. None of this is work that
>> will be merged, rather it is stuff to figure out what we need to
>> know to do the eventual merging correctly and efficiently.
>>
>> Please note that doing that is just the near edge of a large
>> collection of changes that will cascade in many ways to many
>> projects, tools, distros, etc. The people doing this are aware of
>> that, and the relative simplicity (and fairly immediate success) of
>> these experiments is not misleading people into thinking "hey, no
>> big deal". It's a big deal.
>>
>> There's a strategy now (described at the end of the first etherpad
>> listed above) for trimming the nova history to create a thing which
>> is placement. From the first run of that Ed created a github repo
>> and I branched that to eventually create:
>>
>> https://github.com/EdLeafe/placement/pull/2
>>
>> In that, all the placement unit and functional tests are now
>> passing, and my placecat [1] integration suite also passes.
>>
>> That work has highlighted some gaps in the process for trimming
>> history which will be refined to create another interim repo. We'll
>> repeat this until the process is smooth, eventually resulting in an
>> openstack/placement.
> 
> We talked about the github strategy a bit in the placement meeting today
> [1]. Without being involved in this technical extraction work for the
> past few weeks, I came in with a different perspective on the end-game,
> and it was not aligned with what Chris/Ed thought as far as how we get
> to the official openstack/placement repo.
> 
> At a high level, Ed's repo [2] is a fork of nova with large changes on
> top using pull requests to do things like remove the non-placement nova
> files, update import paths (because the import structure changes from
> nova.api.openstack.placement to just placement), and then changes from
> Chris [3] to get tests working. Then the idea was to just use that to
> seed the openstack/placement repo and rather than review the changes
> along the way*, people that care about what changed (like myself) would
> see the tests passing and be happy enough.
> 
> However, I disagree with this approach since it bypasses our community
> code review system of using Gerrit and relying on a core team to approve
> changes at the sake of expediency.
> 
> What I would like to see are the changes that go into making the seed
> repo and what gets it to passing tests done in gerrit like we do for
> everything else. There are a couple of options on how this is done though:
> 
> 1. Seed the openstack/placement repo with the filter_git_history.sh
> script output as Ed has done here [4]. This would include moving the
> placement files to the root of the tree and dropping nova-specific
> files. Then make incremental changes in gerrit like with [5] and the
> individual changes which make up Chris's big pull request [3]. I am
> primarily interested in making sure there are not content changes
> happening, only mechanical tree-restructuring type changes, stuff like
> that. I'm asking for more changes in gerrit so they can be sanely
> reviewed (per normal).
> 
> 2. Eric took a slightly different tack in that he's OK with just a
> couple of large changes (or even large patch sets within a single
> change) in gerrit rather than ~30 individual changes. So that would be
> more like at most 3 changes in gerrit for [4][5][3].
> 
> 3. The 3rd option is we just don't use gerrit at all and seed the
> official repo with the results of Chris and Ed's work in Ed's repo in
> github. Clearly this would be the fastest way to get us to a new repo
> (at the expense of bucking community code review and development process
> - is an exception worth it?).
> 
> Option 1 would clearly be a drain on at least 2 nova cores to go through
> the changes. I think Eric is on board for reviewing options 1 or 2 in
> either case, but he prefers option 2. Since I'm throwing a wrench in the
> works, I also need to stand up and review the changes if we go with
> option 1 or 2. Jay said he'd review them but consider these reviews
> lower priority. I expect we could get some help from some other nova
> cores though, maybe not on all changes, but at least some (thinking
> gibi, alex_xu, sfinucan).
> 
> Any CI jobs would be non-voting while going through options 1 or 2 until
> we get to a point that tests should finally be passing and we can make
> them voting (it should be possible to control this within the repo
> itself using zuul v3).
> 
> I would like to know from others (nova core or otherwise) what they
> would prefer, and if you are a nova core that wants option 1 (or 2) are
> you willing to help review those incremental changes knowing it will be
> a drain - but also realizing that we can't really let option 1 drag on
> while we're doing stein feature development, so ideally this would be
> done before the PTG.
> 
> * Yes I realize I could be reviewing the github pull requests along the
> way, but that's not really how we do code review in openstack.

I think we should use the openstack review system (gerrit) for moving 
the code. We're moving a critical piece of nova to its own repo and I 
think it's worth having the review and history contained in the 
openstack review system.

Using smaller changes that make it easy to see import vs content changes 
might make review faster than fewer, larger changes.

The most important bit of all of this is making sure we don't break 
anything in the process for operators and users consuming nova and 
placement, and ensure the upgrade path from rocky => stein is tested in 
grenade.

The steps I think we should take are:

1. We copy the placement code into the openstack/placement repo and have 
it passing all of its own unit and functional tests.

2. We have a stack of changes to zuul jobs that show nova working but 
deploying placement in devstack from the new repo instead of nova's 
repo. This includes the grenade job, ensuring that upgrade works.

3. When those pass, we merge them, effectively orphaning nova's copy of 
placement. Switch those jobs to voting.

4. Finally, we delete the orphaned code from nova (without needing to 
make any changes to non-placement-only test code -- code is truly orphaned).

-melanie

> [1]
> http://eavesdrop.openstack.org/meetings/nova_scheduler/2018/nova_scheduler.2018-08-27-14.00.log.html#l-74
> [2] https://github.com/EdLeafe/placement
> [3] https://github.com/EdLeafe/placement/pull/3
> [4]
> https://github.com/EdLeafe/placement/commit/e3173faf59bd1453c3800b2bf57c2af8cfde1697
> [5]
> https://github.com/EdLeafe/placement/commit/e984bef8587009378ea430dd1c12ca3e40a3c901
>