Open Stack

On 9/5/2018 10:03 AM, Mohammed Naser wrote:
> On Wed, Sep 5, 2018 at 10:57 AM Matt Riedemann<mriedemos at gmail.com>  wrote:
>> On 9/5/2018 8:47 AM, Mohammed Naser wrote:
>>> Could placement not do what happened for a while when the nova_api
>>> database was created?
>> Can you be more specific? I'm having a brain fart here and not
>> remembering what you are referring to with respect to the nova_api DB.
> I think there was a period in time where the nova_api database was created
> where entires would try to get pulled out from the original nova database and
> then checking nova_api if it doesn't exist afterwards (or vice versa).  One
> of the cases that this was done to deal with was for things like instance types
> or flavours.
> 
> I don't know the exact details but I know that older instance types exist in
> the nova db and the newer ones are sitting in nova_api.  Something along
> those lines?

Yeah that more about supporting online data migrations *within* nova 
where new records were created in the API DB and old records would be 
looked up in both the API DB and then if not found there, in the cell 
(traditional nova DB). But you'd also be running the "nova-manage db 
online_data_migrations" CLI to force the migration of the records from 
the cell DB to the API DB.

With Placement split out of nova, we can't really do that. You could 
point placement at the nova_api DB so it can pull existing records, but 
it would continue to create new records in the nova_api DB rather than 
the placement DB and at some point you have to make that data migration.
Maybe you were thinking something like have temporary fallback code in 
placement such that if a record isn't found in the placement database, 
it queries a configured nova_api database? That'd be a ton of work at 
this point, and if it was something we were going to do, we should have 
agreed on that in YVR several months ago, definitely pre-extraction.

> 
>>> I say this because I know that moving the database is a huge task for
>>> us, considering how big it can be in certain cases for us, and it
>>> means control plane outage too
>> I'm pretty sure you were in the room in YVR when we talked about how
>> operators were going to do the database migration and were mostly OK
>> with what was discussed, which was a lot will just copy and take the
>> downtime (I think CERN said around 10 minutes for them, but they aren't
>> a public cloud either), but others might do something more sophisticated
>> and nova shouldn't try to pick the best fit for all.
> If we're provided the list of tables used by placement, we could considerably
> make the downtime smaller because we don't have to pull in the other huge
> tables like instances/build requests/etc

There are no instances records in the API DB, maybe you mean 
instance_mappings? But yes I get the point.

> 
> What happens if things like server deletes happen while the placement service
> is down?

The DELETE /allocations/{consumer_id} requests from nova to placement 
will fail with some keystoneauth1 exception, but because of our old 
friend @safe_connect we likely won't fail the server delete because we 
squash the exception from KSA:

https://github.com/openstack/nova/blob/0f102089dd0b27c7d35f0cbba87332414032c0a4/nova/scheduler/client/report.py#L2069

However, you'd still have allocations in placement against resource 
providers (compute nodes) for instances that no longer exist, which 
means you're available capacity for scheduling new requests is 
diminished until those bogus allocations are purged from placement, 
which will take some scripting.

In other words, not good things.

> 
>> I'm definitely interested in what you do plan to do for the database
>> migration to minimize downtime.
> At this point, I'm thinking turn off placement, setup the new one, do
> the migration
> of the placement-specific tables (this can be a straightforward documented task
> OR it would be awesome if it was a placement command (something along
> the lines of `placement-manage db import_from_nova`) which would import all
> the right things

You wouldn't also stop nova-api while doing this? Otherwise you're going 
to get into the data/resource tracking mess described above which will 
require some post-migration cleanup scripting.

> 
> The idea of having a command would be*extremely*  useful for deployment tools
> in automating the process and it also allows the placement team to selectively
> decide what they want to onboard?
> 
> Just throwing ideas here.
> 


-- 

Thanks,

Matt