Open Stack

Hi all,

I just wanted to send out a quick status report on what work is in
progress in Mitaka and what work is still outstanding.  This is not
inclusive of all work that has occurred this cycle.

There are some reviews up from CERN to create flavor tables in the
nova_api database and begin the migration of flavor data out of the nova
db, which will be in a cell soon, up to the top level db.  Since we are
no longer adding soft-delete capabilities to new database tables we will
lose the ability to soft-delete flavors after the migration.  This
wouldn't be a big issue except that the Nova API allows a user to
request a deleted flavor.  There is a thread with details at
http://lists.openstack.org/pipermail/openstack-dev/2016-January/083518.html
but the outcome was that we will retain the use case that viewing
deleted flavors served but implement it in a better manner.  Related
reviews are:

https://review.openstack.org/#/c/201606/ flavor table in api db
https://review.openstack.org/265282 Spec for allowing flavors to be hard
deleted
https://review.openstack.org/#/c/213041 migration of flavor data to api
db

The next big body of work is about updating the instance boot process to
what it needs to look like in a cells world.  There are a few chunks of
work here.  The first is to create a RequestSpec object and persist it
before the Instance object has been created and saved to the database. 
This is necessary because with cells the Instance will not be created
until after scheduling has picked a cell/host but the API needs to
return a response before that happens so the details of the boot request
need to be saved before instance creation.  Related reviews:

https://review.openstack.org/#/c/254434/ Build filter_properties earlier
in boot process.
https://review.openstack.org/258628 persist request spec during boot

We also need to store some instance details that do not fit within the
RequestSpec but are necessary to satisfy an instance show/list.  These
additional fields will go in a new BuildRequest object and db table so
when the API needs to display an instance which does not yet exist in a
cell it will create the instance from the BuildRequest/RequestSpec. 
Related reviews:

https://review.openstack.org/#/c/263926/ (WIP)
https://review.openstack.org/#/c/263927/ (WIP)

We also need some cells in which to place these instances, and to map
the instances once scheduled.  There is some very early work in the
following reviews:

https://review.openstack.org/#/c/263925 populate null instance mapping
during boot
https://review.openstack.org/#/c/267828 update instance mapping after
scheduling

And Melanie Witt has agreed to work on a method for bootstrapping the
current nova database and message queue into the first cell.  This
process will also map the current computes to that cell.

The final part of the boot process in motion is something we're calling
cell0.  This will be the holding area for instances that can not be
scheduled to a compute within a cell.  Cell0 will essentially just be a
database to hold these instances until they are deleted by a user.  Mark
Doffman and Chuck Carmack have agreed to work on cell0.  Related review:

https://review.openstack.org/#/c/267827 map instance to cell0 on
scheduling failure (WIP)

There is also some testing work to be done.  Chuck Carmack has been
pushing forward on getting Tempest to a point where it's testing SSH
access to an instance in the cellsv1 job.  Testing SSH access was lost
when the basic Devstack exercises stopped being run.  The Tempest
test(s) that would exercise this behavior are blacklisted for cells
because they rely on security groups created by the test.  Cellsv1 has
never supported security groups so that does not work.  Related review:

https://review.openstack.org/#/c/225199/ Add a config option to disable
security group usage

And some miscellaneous that will become relevant once we're looking to
support multiple cells, i.e. not in Mitaka:

https://review.openstack.org/#/c/161906/ db connection switching

Areas of opportunity:  There are a number of WIP reviews listed above
that I have pushed up but would be happy to share if someone was
interested in getting involved.  Additionally we could use some people
looking at grenade to ensure that we're testing all of the big changes
coming.  Things like the addition of the new nova_api db, the creation
of the first cell and mapping of hosts in that cell, and new instance
mappings being created.  Standard functional testing will cover parts of
these but it may not be possible to ensure things like host mappings
being created properly or other db changes that will be occurring.  So
some checks that confirm that the database looks like it should could be
useful.

If you have any interest in getting involved or following the progress
please reach out to me or attend a weekly meeting
https://wiki.openstack.org/wiki/Meetings/NovaCellsv2.