Open Stack

On 12/16/2016 3:20 AM, Sylvain Bauza wrote:
>
>
> Le 16/12/2016 03:53, Matt Riedemann a écrit :
>> A few of us have talked about writing a command to tell when you're
>> ready to upgrade (restart services with new code) for Ocata because we
>> have a few things landing in Ocata which are going from optional to
>> required, namely cells v2 and the placement API service.
>>
>> We already have the 030 API DB migration for cells v2 which went into
>> the o-2 milestone today which breaks the API DB schema migrations if you
>> haven't run at least 'nova-manage cell_v2 simple_cell_setup'.
>>
>> We have noted a few times in the placement/resource providers
>> discussions the need for something similar for the placement API, but
>> because that spans multiple databases (API and cell DBs), and because it
>> involves making sure the service is up and we can make REST API requests
>> to it, we can't do it in just a DB schema migration.
>>
>> So today dansmith, jaypipes, sdague, edleafe and myself jumped on a call
>> to go over some notes / ideas being kicked around in an etherpad:
>>
>> https://etherpad.openstack.org/p/nova-ocata-ready-for-upgrade
>>
>
> Thanks for having done that call and having written that etherpad, that
> clarifies a lot of open questions I had in mind without having time to
> kick them off on IRC.
>
>> We agreed on writing a new CLI outside of nova-manage called nova-status
>> which can perform the upgrade readiness check for both cells v2 and the
>> placement API.
>>
>> For cells v2 it's really going to check basically the same things as the
>> 030 API DB schema migration.
>>
>> For the placement API, it's going to do at least two things:
>>
>> 1. Try and make a request to / on the placement API endpoint from the
>> service catalog. This will at least check that (a) the placement
>> endpoint is in the service catalog, (b) nova.conf is configured with
>> credentials to make the request and (c) the service is running and
>> accepting requests.
>>
>> 2. Count the number of resource_providers in the API DB and compare that
>> to the number of compute_nodes in the cell DB and if there are fewer
>> resource providers than compute nodes, it's an issue which we'll flag in
>> the upgrade readiness CLI. This doesn't necessarily mean you can't
>> upgrade to Ocata, it just means there might be fewer computes available
>> for scheduling once you get to Ocata, so the chance of rebuilds and
>> NoValidHost increases until the computes are upgraded to Ocata and
>> configured to use the placement service to report
>> inventories/usage/allocations for RAM, CPU and disk.
>>
>
> Jus to make it clear, that CMD tool is mostly giving you a pre-flight
> status on whether you could upgrade to Ocata, right?
>
> I mean, operators can use that tool for getting a light, either green,
> orange or red telling them if they'll encounter either no issues,
> possible ones or definitely errors, right?

Correct, that's the intent. You're the second person to bring up colors 
as status indicators though (red, yellow, green for US traffic signals 
was the other). I'm going to do something like success/warning/error 
instead of colors to avoid confusion (I think they have blue traffic 
signals in Japan?). And corresponding return codes (0 = success, 1 = 
warnings, 2 = something failed).

>
> If so, I think that's a nice tool to have that we could gradually
> improve to make it convenient for people wanting to know if they miss
> something before upgrading.
>
>> That 2nd point is important because we also agreed to make the filter
>> scheduler NOT fallback to querying the compute_nodes table if there are
>> no resource providers available from the placement API. That means when
>> the scheduler gets a request to build or move a server, it's going to
>> query the placement API for possible resource providers to serve the
>> CPU/RAM/disk requirements for the build request spec and if nothing is
>> available it'll result in a NoValidHost failure. That is a change in
>> direction from a fallback plan we originally had in the spec here:
>>
>> https://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/resource-providers-scheduler-db-filters.html#other-deployer-impact
>>
>>
>
> I just noticed something wrong with that section since we agreed when
> reviewing that spec that we won't remove the legacy CPU/RAM/MEM filters
> in Pike since the CachingScheduler will still use them.
>
> We'll just default to disable them for the FilterScheduler, but anyone
> could just enable them back if they want to use the CachingScheduler.

OK, we need to update that spec anyway to remove the provision about the 
fallback mechanism in the filter scheduler.

I'll amend the spec if someone else doesn't get to it first.

>
>> We're changing direction on that because we really want to make the
>> placement service required in Ocata and not delay it's usage for another
>> release, because as long as it's optional people are going to avoid
>> deploying and using it, which pushes us further out from forward
>> progress around the scheduler, placement service and resource tracker.
>>
>
> I fully agree with that goal. We already communicated that we will
> require the Placement service in Ocata in the Newton release notes so
> there is nothing really new here.
>
>
>> Regarding where this new CLI lived, and how it's deployed, and when it's
>> called, we had discussed a few options there, even talking about
>> splitting it out into it's own pip-installable package. We have a few
>> options but we aren't going to be totally clear on that until we get the
>> POC code written and then try to integrate it into grenade, so we're
>> basically deferring that discussion/decision for now. Wherever it is, we
>> know it needs to be run with the Ocata code (since that's where it's
>> going to be available), and after running the simple_cell_setup
>> command), and it needs to be run before restarting services with the
>> Ocata code. I'm not totally sure if it needs to be run after the DB
>> migrations or not, maybe Dan can clarify, but we'll sort it out for sure
>> when we integrate with grenade.
>>
>> Anyway, the POC is started here:
>>
>> https://review.openstack.org/#/c/411517/
>>
>
> Cool, count me on the reviewers list.
>
> Anyhow, I'll start writing the scheduler change that will call the
> placement API soon-ish (hopefully, beginning of next week it will be a
> thing) and I'll make sure that we default deploying the placement
> service by devstack as a Depends-On (like we discussed yesterday on IRC).
>
> -Sylvain
>
>> I've got the basic framework in place and there is a patch on top that
>> does the cells v2 check. I plan on working on the placement API checks
>> tomorrow.
>>
>> If you've read this far, congratulations. This email is really just
>> about communicating that things are happening because we have talked
>> about the need for this a few times, but hadn't hashed it out yet.
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 

Thanks,

Matt Riedemann