[openstack-dev] [nova] [placement] extraction (technical) update

Matt Riedemann mriedemos at gmail.com
Tue Aug 28 15:47:12 UTC 2018


On 8/28/2018 6:20 AM, Chris Dent wrote:
> On Mon, 27 Aug 2018, melanie witt wrote:
> 
>> I think we should use the openstack review system (gerrit) for moving 
>> the code. We're moving a critical piece of nova to its own repo and I 
>> think it's worth having the review and history contained in the 
>> openstack review system.
> 
> This seems a reasonable enough strategy, in broad strokes. I want to
> be sure that we're all actually in agreement on the details, as
> we've had a few false starts and I think some of the details are
> getting confused in the shuffle and the general busy-ness in progress.
> 
> Is anyone aware of anyone who hasn't commented yet that should? If
> you are, please poke them so we don't surprise them.
> 
>> Using smaller changes that make it easy to see import vs content 
>> changes might make review faster than fewer, larger changes.
> 
> I _think_ we ought to be able to use the existing commits from the
> runs-throughs-to-passing-tests already done, but if we use the
> strategy described below it doesn't really matter: the TDD approach
> (after fixing paths and test config) is pretty fast.
> 
>> The most important bit of all of this is making sure we don't break 
>> anything in the process for operators and users consuming nova and 
>> placement, and ensure the upgrade path from rocky => stein is tested 
>> in grenade.
> 
> This is one of the areas where pretty active support from all of
> nova will be required: getting zuul, upgrade paths, and the like
> clearly defined and executed.
> 
>> The steps I think we should take are:
>>
>> 1. We copy the placement code into the openstack/placement repo and 
>> have it passing all of its own unit and functional tests.
> 
> To break that down to more detail, how does this look?
> (note the ALL CAPS where more than acknowledgement is requested)
> 
> 1.1 Run the git filter-branch on a copy of nova
>      1.1.1 Add missing files to the file list:
>            1.1.1.1 .gitignore
>            1.1.1.2 # ANYTHING ELSE?

Unless I were to actually run the git filter-branch tooling to see what 
is excluded from the new repo, I can't really say what is missing at 
this time. I assume it would be clear during review - which is why I'm 
asking that we do this stuff in gerrit where we do reviews.

> 1.2 Push -f that thing, acknowledge to be broken, to a seed repo on github
>      (ed's repo should be fine)
> 1.3 Do the repo creation bits described in
>      https://docs.openstack.org/infra/manual/creators.html
>      to seed openstack/placement
>      1.3.1 set zuul jobs. Either to noop-jobs, or non voting basic
>      func and unit # INPUT DESIRED HERE

Agree. As I said to gibi elsewhere in this thread, I would hold off on 
adding a tempest-full job to the repo until we're at the end.

> 1.4 Once the repo exists with some content, incrementally bring it to
>      working
>      1.4.1 Update tox.ini to be placement oriented
>      1.4.2 Update setup.cfg to be placement oriented
>      1.4.3 Correct .stesr.conf
>      1.4.4 Move base of placement to "right" place
>      1.4.5 Move unit and functionals to right place
>      1.4.6 Do automated path fixings
>      1.4.7 Set up translation domain and i18n.py corectly
>      1.4.8 Trim placement/conf to just the conf settings required
>            (api, base, database, keystone, paths, placement)
>      1.4.9 Remove database files that are not relevant (the db api is
>            not used by placement)
>      1.4.10 Fix the Database Fixture to be just one database
>      1.4.11 Disable migrations that can't work (because of
>             dependencies on nova code, 014 and 030 are examples)
>             # INPUT DESIRED HERE AND ON SCHEMA MIGRATIONS IN GENERAL
>      1.4.12 Incrementally get tests working
>      1.4.13 Fix pep8
> 1.5 Make zuul pep, unit and functional voting
> 1.6 Create tools for db table sync/create
> 1.7 Concurrently go to step 2, where the harder magic happens.
> 1.8 Find and remove dead code (there will be some).
> 1.9 Tune up and confirm docs
> 1.10 Grep for remaining "nova" (as string and spirit) and fix
> 
> 
> Item 1.4.12 may deserve some discussion. When I've done this the
> several times before, the strategy I've used is to be test driven:
> run either functional or unit tests, find and fix one of the errors
> revealed, commit, move on.
> 
> This strategy has worked very well for me because of the "test
> driven" part, but I'm hesitant to do it if reviewers are going to
> get to a patch and say "why didn't you also change X?" The answer to
> that question is "because this is incremental and test driven and
> the tests didn't demand that change (yet)". Sometimes that will mean
> that things of the same class of change are in different commits.
> 
> Are people okay with that and willing to commit to being okay with
> that answer in reviews? To some extent we need to have some faith on
> the end result: the tests work. If people are not okay with that, we
> need the people who are not to determine and prove the alternate
> strategy. I've had this one work and work well.

Seems reasonable to me. But to be clear, if there are 70 failed tests, 
are you going to have 70 separate patches? Or this is just one of those 
things where you start with 70, fix something, get down to 50 failed 
tests, and iterate until you're down to all passing. If so, I'm OK with 
that. It's hard to say without knowing how many patches get from 70 
failures to 0 and what the size/complexity of those changes is, but 
without knowing I'd default to the incremental approach for ease of review.

> 
> Please help to refine the above, thank you.
> 
>> 2. We have a stack of changes to zuul jobs that show nova working but 
>> deploying placement in devstack from the new repo instead of nova's 
>> repo. This includes the grenade job, ensuring that upgrade works.
> 
> If we can make a list for this (and the subsequent) major items that
> is as detailed as I've made for step 1 above, I think that will help
> us avoid some of the confusion and frustration that comes up. I'm
> neither able nor willing to be responsible for creating those lists
> for all these points, but very happy to help.

Grenade uses devstack so once we have devstack on master installing (and 
configuring) placement from the new repo and disable installing and 
configuring it from the nova repo, that's the majority of the change I'd 
think.

Grenade will likely need a from-rocky script to move any config that is 
necessary, but as you already noted below, if the new repo can live with 
an existing nova.conf, then we might not need to do anything in grenade 
since placement from the new repo (in stein) could then run with 
nova.conf created for placement from the nova repo (in rocky).

> 
>> 3. When those pass, we merge them, effectively orphaning nova's copy 
>> of placement. Switch those jobs to voting.
>>
>> 4. Finally, we delete the orphaned code from nova (without needing to 
>> make any changes to non-placement-only test code -- code is truly 
>> orphaned).
> 
> In case you missed it, one of the things I did earlier in the
> discussion was make it so that the wsgi script for placement defined
> in nova's setup.cfg [1] could:
> 
> * continue to exist
> * with the same name
> * using the nova.conf file
> * running the extracted placement code
> 
> That was easy to do because of the work over the last year or so
> that has been hardening the boundary between placement and nova, in
> place. I've been assuming that maintaining the option to use
> original conf file is a helpful trick for people. Is that the case?

Yes, as noted above, that might mean we don't need any changes in 
grenade, but I can't foretell what will be needed until we start to 
actually work on the devstack/grenade changes and see what fails and 
then iterate on fixing those issues - exactly what you're doing with 
fixing the in-tree unit/functional tests.

> 
> Thanks.
> 
> [1] 
> https://review.openstack.org/#/c/596291/3/nova/api/openstack/placement/wsgi.py 
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 

Thanks,

Matt



More information about the OpenStack-dev mailing list