Open Stack

We had two design summit sessions for cells v2 at the Ocata summit in 
Barcelona. The full etherpads are here:

https://etherpad.openstack.org/p/ocata-nova-summit-cellsv2-scheduler
https://etherpad.openstack.org/p/ocata-nova-summit-cellsv2-quotas

In the first session we mostly talked about the work items needed to 
support multiple cells.

Scheduler interaction
---------------------

Andrew Laski had the spec up for this and there was some POC code 
started, which Dan Smith is picking up. The main idea here is to create 
the BuildRequest in nova-api, then call a new conductor method which 
calls the scheduler to pick a host, and then conductor creates the 
instance in the cell mapped to that host, and then conductor deletes the 
BuildRequest. Once the instance is in that cell, it doesn't move out of 
that cell via reschedule/rebuild/migration/evacuate. We might add 
support for that later, but it's not something supported in the initial 
cells v2 scheduler work.

Cell0
-----

Cell0 is the special database (using the same cell DB schema) where 
instances that failed to build go to die. In Newton this was optional 
but the API will pull instances from it when listing instances. However, 
we aren't populating it on schedule failure yet so that's work that 
needs to be done for Ocata. So far there are no patches up for this yet, 
but it will most likely be worked in with the scheduler interaction series.

The listing instances problem
-----------------------------

In order to list instances we need to be able to page across cells, 
which is going to be expensive for multiple large cells. Long-term we 
want to use searchlight for this, but in the short term we're going to 
do a simple merge sort in python. We also need to get the fixes in to 
restrict the filter/sort keys for listing instances, Alex Xu and Kevin 
Zheng are working on that. There are open questions on the long-term 
plans of how things are going to work with searchlight, and Chris Dent 
said he was interested in working on that. As a start, the searchlight 
team has started reporting bugs against nova for gaps in the 
notifications that nova sends out compared to the REST API. Balazs 
Gibizer (gibi) who runs the notifications subteam in nova has already 
started triaging those bugs.

Testing
-------

CI testing the multi-cell configuration should be relatively 
straight-forward with a multinode job. We can have one node contain the 
'control' bits like the API, conductor, scheduler, API/cell0/cell DBs, 
along with a nova-compute service, and then another node that is just 
running nova-compute. I've volunteered to work on that.

Dan and I have also started working on a series of changes to make 
cellsv2 required in the Ocata CI jobs:

https://review.openstack.org/#/q/topic:ocata-requires-cellsv2

This consists of a nova database migration that fails if you haven't 
created the cell0 database and run the simple_cell_setup command.

Upgrade
-------

Today in non-cells deployments we say to upgrade conductors first, then 
API and finally computes. With cells v2 we want to do rolling upgrades 
of the cells, which means upgrading conductors and then computes in the 
cells, and finally the API. This allows you to only enable the latest 
features in the API when the cells are all upgraded and ready to handle 
those requests. This poses a bit of a problem for our CI tooling with 
devstack/grenade though as we upgrade and start nova-api first. That's 
going to require some changes as we get into the multi-cell testing 
mentioned above.

Quotas
------

In the second design summit session on cells v2 we spent the entire time 
talking about quotas, and thinking about ways to potentially redo the 
quotas design when moving those to the API database.

Melanie Witt has a spec up which proposes that instead of doing the 
reserve (API), do work, commit/rollback (compute) model, we move commits 
to the API and have a process of (1) do work and then (2) reserve and 
commit.

While talking about that in the session, there was some brainstorming on 
doing limit checks differently in the API. Basically, do away with 
reservations, do a quick DB query to check quota before an operation 
begins, and if it's OK go forward. There will be races as tenants reach 
the end of quota limits, but maybe this is OK. The upside is we 
shouldn't get out of sync (which has been a problem for operators to 
deal with), but there is a potential to race for the last usages which 
might lead to overconsumption of resources.

There are some open questions around the 'fast and loose' approach like 
are there things we need to count which aren't in the API database, and 
can we do this efficiently for things that nova doesn't track in it's 
database, like floating/fixed IPs in neutron?

-- 

Thanks,

Matt Riedemann