[openstack-dev] [nova] Belated nova newton midcycle recap (part 2)
mriedem at linux.vnet.ibm.com
Tue Aug 2 02:15:46 UTC 2016
Starting from where I accidentally left off:
* Vendor metadata reboot
We agreed that we still wanted mikal to keep working on this so we can
keep a timeline for removing the deprecated dynamic vendor data classloader.
The API change was merged last week:
There are still some testing and documentation changes left.
* Microversion API testing in Tempest
We talked about the current state of getting changes into Tempest for
Nova microversions and how a bunch of changes were up for review at the
same time for backfill some schema responses, like for 2.3 and 2.26.
I already posted what I thought we had agreed on at the midcycle:
But there is some disagreement about how I tried to write that up in the
Tempest docs so we're still trying to hash out this policy:
* Placement API for resource providers
Jay's personal goal for Newton is for the resource tracker to be writing
inventory and allocation data via the placement API. We want to get the
data writing into the placement API in Newton so we can start using it
There are some spec amendments up for resource providers, at least one
has merged, and the initial placement API change merged today:
We talked about supporting dynamic resource classes for Ironic use cases
which is a stretch goal for Nova in Newton. Jay has a spec for that here:
There is a lot more detail in the etherpad and honestly Jay Pipes or Jim
Rollenhagen would be better to summarize what came out of this at the
midcycle and what's being worked on for dynamic resource classes right now.
We talked about a separate placement API database but decided this
should be optional to avoid forcing yet another nova database on
deployers in a couple of releases. This would be available for deployers
to use to avoid some future upgrade pain when the placement service is
split out from Nova, but if not configured it will default to the API
database for the placement API. There are a bunch more details and
discussion on that in this thread that Chris Dent started after the
* nova/cinder interlock
A few of us called into the cinder midcycle hangout to talk through a
few ongoing efforts between projects.
John Griffith has some POC code up that adds a new set of APIs to Cinder
which consolidates the os-reserve, os-initialize_connection and
os-attach APIs into a single API along with changes to cinderclient and
nova to use the APIs. This is to close the gap on some long-standing
race issues between nova and cinder volume attach operations and will
feed into the volume multi-attach work as cinder will be storing the
attachment information differently so we can detach properly. I need to
fix up my devstack(-gate) change to test the entire stack, and John
Griffith was going to write a spec for Cinder for the new APIs.
John Garbutt and I had a TODO to review Walter Boring's bug fix /
cleanup nova-api change to stop doing state checking from the API and
let Cinder handle that in os-reserve:
There is some other related work to that but it gets a bit more
complicated in the boot from volume and live migration cases. John
Griffith was also going to check with some other Cinder storage
providers like Ceph/FC to make sure these changes would be OK for them,
and to check on live migration testing (PureStorage is working on
multinode live migration test coverage for their third party CI using
Matt Treinish also helped sort out some issues with getting a cinder
multi-backend job in the gate that Scott D'Angelo has been working on.
There is a series of changes to project-config, devstack and Tempest to
get this testing working so we can test things like volume
retype/migration and swap-volume in the gate with an LVM backend. The
devstack change should just be a pass-through config to Tempest from the
job rather than the existing approach.
* nova/neutron interlock
Carl Baldwin was at the meetup so we mostly talked about routed networks
and the deferred IP allocation change he's been working on:
This is a step toward using routed networks before we have the full
proper scheduling in place with resource providers and the placement
API. We talked through some issues with build failures and rescheduling
which right now will just reschedule until failure, but Carl has some
changes to detect fixed IP allocation on the wrong host and fail, which
Nova can then handle and abort the build rather than reschedule. We can
work that in as a bug fix though once we get the initial change in to
support deferred IP allocation.
We also talked a bit about live migration with Neutron. There has been a
fix up for live migration + DVR since Mitaka:
It's a bit of a hacky workaround but the longer term solution that we
all want ( https://review.openstack.org/#/c/309416 ) is not going to be
in Newton and will need discussion at the Ocata summit in Barcelona
(John Garbutt was going to work with the Neutron team on preparing for
the summit on that one). So we agreed to go with Swami's DVR fix but it
needs to be rebased (which still hasn't happened since the midcycle).
* Host capabilities
A lot of the discussion for this is happening in the
At the midcycle we agreed that we didn't want things like QoS flavor
extra_specs going to the placement API ever, those just go to the
compute that is picked by the scheduler.
We agreed that we want a common os-capabilities library that other
projects like Ironic can use (not an oslo library). This could be
started in Newton and can store the base enums that we've already
defined - it will be a catalog for standardized capabilities. We also
want to start extracting capability-type things out of
flavor.extra_specs into a new flavor.capabilities field (which would
happen when lazy-loading the field). We'd do something similar for
ImageMeta.properties. The library would have a routine that takes a
flavor extra_spec and returns a capability. Dan Smith was pretty adamant
about wanting to be involved in this. We also said that the placement
API wouldn't take scheduler_hints.
* Mix and match resource federation
This is an item that's come up at the past few summits but we finally
got into some deep discussion about it, the spec is here:
This is about Nova talking to different Cinder vendors/sites. There are
several problems with Nova managing this though so we agreed that the
team wanting this would be better served writing a separate proxy
service which can listen to notifications from nova and cinder and
maintain a cache/mapping of volumes to the sites they exist in (sort of
like cells). The proxy service could then also be configured however
they want, including API policy. The only change to Nova then should be
configuration only, which is telling Nova which service catalog entry to
use for volumes, which would point to the proxy. There was an open
question about how this would work in the cases where Cinder calls back
to Nova APIs like swap-volume during volume retype/migration, but that's
only for certain Cinder volume drivers (GlusterFS, NFS, other FS-style
drivers). George and Kristi were going to investigate that.
* Future of compute host metrics monitoring in Nova
We currently have a CPU metrics monitor in Nova for scheduling. There
was a blueprint to add memory bandwidth monitoring and another blueprint
to expose these pluggable monitors and their data from the database via
the nova-manage CLI. Projects like Watcher use these monitors.
We don't want Nova to be in the business of metrics gathering and
monitoring, there are several other tools available for doing that. What
Watcher wants/needs is an ability to influence placement decisions (all
other things equal between hosts). So the idea came up to provider a
weight field on resource providers via the placement API so that Watcher
could use it's time series database (fed from external monitors) to
adjust weights on resource providers as necessary (keeping in mind
compute hosts are resource providers too).
So we plan to deprecate the monitor metric plugins as a signal that we
don't want that monitoring code in nova long-term. Given we don't have a
solid replacement defined yet, this would be an open-ended deprecation
as an indication that new services/deployments shouldn't be built on it
and people should help work on alternatives, like the resource provider
While on the topic of Watcher (and Joe Cropper was with us), we talked
about another requirement to be able to pass a list of hosts to the
migration APIs (live and cold) so that admins don't have to use force
hosts. There was some agreement in the room that this is generally a
useful use case / feature, not really only for Watcher. However, there
was some concern about opening up the API to pass lists of hosts which
could creep into other APIs and then the Nova API eventually becoming a
proxy for the placement API. This is an Ocata spec though so more
discussion will happen in the spec once we open those up.
* Functional devstack (nova-dsvm-integration job)
We could use a dsvm-integration job which has a devstack running but has
nova in-tree tests that aren't Tempest. This could be useful for testing
things that rely on libvirt/qemu like the imagebackend refactor series
related to libvirt storage pools.
Since Sean Dague isn't available to work on this right now we punted.
* Next steps for capabilities/policy exposure in the API
We're going to start small with this in Ocata with a couple of APIs and
see how things work, like os-attach-interfaces (the Ironic virt driver
doesn't support that). Then when we're happy with it we can build on it
for things that don't exist yet, like live resize. We have to keep in
mind that capabilities are different based on context, i.e. what's
available in this cloud vs what are you capable (allowed) to do in this
We said this wouldn't be using microversions but would probably be more
like the metadata API which uses dates.
There were also questions about how to model the capabilities, which
ties into host capabilities, which is all Ocata summit discussion to be had.
* Live resize
Now that I've mentioned live resize, I'll say we agreed it was something
we should support (in Ocata) but it's dependent on the
capabilities/policy exposure in the API. We also said that it would be
restricted to live resize on the same host and with existing flavors,
i.e. you can't pass random values like CPU/RAM/Disk for the resize - it
has to be a flavor. And you can't resize down.
* Moving docs from openstack-manuals into the nova repo
There is a related mailing list thread on this:
There should probably be a cross-project session on this at the summit.
While we understand the issue, we don't think this is something that we
really want moved to Nova, for a few reasons:
- We don't have anyone in nova that cares enough about docs to own/drive
this so it's hard to make traction (unlike the api-ref push that Sean
Dague has done).
- Review time in docs is ~1 day, review time in Nova is ~19 days. So
moving the docs into Nova actually slows things down for the docs team.
- As an example, Nova doesn't care about writing a SLES install guide.
There was general sentiment that the docs team should feel free to work
on this for other projects where it works, but doing it in nova will
probably make things worse.
The TODO on this one was Sean Dague said he'd reply to the above thread
stating as much.
* Migration state machine
Timofey Durakov is proposing a state machine for orchestration of things
like live migration which would be heavily tied to the idea of a tasks
engine. There was discussion about what problem(s) we're actually trying
to solve (of which there are many) and what concrete steps we can start
taking now. We agreed that Timofey should start defining what our state
transitions are right now for various operations like live/cold
migration and resize. This would be "on paper", i.e. a spec. Then start
making the code follow the state machine which would probably be new RPC
calls so we don't try to retrofit the existing code to handle this
(start clean). Since Ironic people were at the midcycle they also
pointed out they have are using the automaton library and maybe we could
follow some of what they are doing.
* Hyper-v cluster driver
In the existing proposal hyper-v changes the instance host underneath
nova and then the virt driver has to reconcile the changes in the
resource tracker. This is problematic for a few reasons, but major ones
are the fact that hyper-v has to anticipate nova's data structures,
placement logic and the request spec used to build the instance. We also
plan to eventually move claims from the compute into the scheduler so
the virt driver wouldn't be able to adjust claims after an out-of-band
migration. While not great, an alternative would be for hyper-v (the
hypervisor) to have a nova plugin which would call the Nova REST API and
tell it where to move an instance.
* Cold migration API improvements
Takashi Natsume has several specs for Ocata to make improvements to the
cold migration API that exist in the live migration API, like force
hosts, check destination host and abort support. We agreed that these
are a good idea since operators would like to fallback to similar
operations with cold migration if live migration fails.
* The rest
I had to leave after lunch on Thursday to catch my flight home so I
missed some of the random topics discussed that afternoon. The notes are
in the etherpad but I'm not sure if there was anything major that came
up which needs a mention here. If I'm wrong and I just hurt your
feelings, I'm sorry, please reply and recap any discussion/decisions made.
In closing, it was a good time, there was a ton of content and a lot of
people showed up. I'm a bit scared about how much work we have left to
do in Newton and we've already started piling on for Ocata, but that's
the way it is. We definitely have a lot to build on for the upcoming
summit in October though.
More information about the OpenStack-dev