Open Stack

Starting from where I accidentally left off:

* Vendor metadata reboot

We agreed that we still wanted mikal to keep working on this so we can 
keep a timeline for removing the deprecated dynamic vendor data classloader.

The API change was merged last week:

https://review.openstack.org/#/c/317739/

There are still some testing and documentation changes left.

* Microversion API testing in Tempest

We talked about the current state of getting changes into Tempest for 
Nova microversions and how a bunch of changes were up for review at the 
same time for backfill some schema responses, like for 2.3 and 2.26.

I already posted what I thought we had agreed on at the midcycle:

http://lists.openstack.org/pipermail/openstack-dev/2016-July/099860.html

But there is some disagreement about how I tried to write that up in the 
Tempest docs so we're still trying to hash out this policy:

https://review.openstack.org/#/c/346092/

* Placement API for resource providers

Jay's personal goal for Newton is for the resource tracker to be writing 
inventory and allocation data via the placement API. We want to get the 
data writing into the placement API in Newton so we can start using it 
in Ocata.

There are some spec amendments up for resource providers, at least one 
has merged, and the initial placement API change merged today:

https://review.openstack.org/#/c/329149/

We talked about supporting dynamic resource classes for Ironic use cases 
which is a stretch goal for Nova in Newton. Jay has a spec for that here:

https://review.openstack.org/#/c/312696/

There is a lot more detail in the etherpad and honestly Jay Pipes or Jim 
Rollenhagen would be better to summarize what came out of this at the 
midcycle and what's being worked on for dynamic resource classes right now.

We talked about a separate placement API database but decided this 
should be optional to avoid forcing yet another nova database on 
deployers in a couple of releases. This would be available for deployers 
to use to avoid some future upgrade pain when the placement service is 
split out from Nova, but if not configured it will default to the API 
database for the placement API. There are a bunch more details and 
discussion on that in this thread that Chris Dent started after the 
midcycle:

http://lists.openstack.org/pipermail/openstack-dev/2016-July/100302.html

* nova/cinder interlock

A few of us called into the cinder midcycle hangout to talk through a 
few ongoing efforts between projects.

John Griffith has some POC code up that adds a new set of APIs to Cinder 
which consolidates the os-reserve, os-initialize_connection and 
os-attach APIs into a single API along with changes to cinderclient and 
nova to use the APIs. This is to close the gap on some long-standing 
race issues between nova and cinder volume attach operations and will 
feed into the volume multi-attach work as cinder will be storing the 
attachment information differently so we can detach properly. I need to 
fix up my devstack(-gate) change to test the entire stack, and John 
Griffith was going to write a spec for Cinder for the new APIs.

John Garbutt and I had a TODO to review Walter Boring's bug fix / 
cleanup nova-api change to stop doing state checking from the API and 
let Cinder handle that in os-reserve:

https://review.openstack.org/#/c/315789/

There is some other related work to that but it gets a bit more 
complicated in the boot from volume and live migration cases. John 
Griffith was also going to check with some other Cinder storage 
providers like Ceph/FC to make sure these changes would be OK for them, 
and to check on live migration testing (PureStorage is working on 
multinode live migration test coverage for their third party CI using 
iscsi/fibrechannel).

Matt Treinish also helped sort out some issues with getting a cinder 
multi-backend job in the gate that Scott D'Angelo has been working on. 
There is a series of changes to project-config, devstack and Tempest to 
get this testing working so we can test things like volume 
retype/migration and swap-volume in the gate with an LVM backend. The 
devstack change should just be a pass-through config to Tempest from the 
job rather than the existing approach.

* nova/neutron interlock

Carl Baldwin was at the meetup so we mostly talked about routed networks 
and the deferred IP allocation change he's been working on:

https://review.openstack.org/#/c/299591/

This is a step toward using routed networks before we have the full 
proper scheduling in place with resource providers and the placement 
API. We talked through some issues with build failures and rescheduling 
which right now will just reschedule until failure, but Carl has some 
changes to detect fixed IP allocation on the wrong host and fail, which 
Nova can then handle and abort the build rather than reschedule. We can 
work that in as a bug fix though once we get the initial change in to 
support deferred IP allocation.

We also talked a bit about live migration with Neutron. There has been a 
fix up for live migration + DVR since Mitaka:

https://review.openstack.org/#/c/275073

It's a bit of a hacky workaround but the longer term solution that we 
all want ( https://review.openstack.org/#/c/309416 ) is not going to be 
in Newton and will need discussion at the Ocata summit in Barcelona 
(John Garbutt was going to work with the Neutron team on preparing for 
the summit on that one). So we agreed to go with Swami's DVR fix but it 
needs to be rebased (which still hasn't happened since the midcycle).

* Host capabilities

A lot of the discussion for this is happening in the 
ResourceProviderTags thread:

http://lists.openstack.org/pipermail/openstack-dev/2016-July/099032.html

At the midcycle we agreed that we didn't want things like QoS flavor 
extra_specs going to the placement API ever, those just go to the 
compute that is picked by the scheduler.

We agreed that we want a common os-capabilities library that other 
projects like Ironic can use (not an oslo library). This could be 
started in Newton and can store the base enums that we've already 
defined - it will be a catalog for standardized capabilities. We also 
want to start extracting capability-type things out of 
flavor.extra_specs into a new flavor.capabilities field (which would 
happen when lazy-loading the field). We'd do something similar for 
ImageMeta.properties. The library would have a routine that takes a 
flavor extra_spec and returns a capability. Dan Smith was pretty adamant 
about wanting to be involved in this. We also said that the placement 
API wouldn't take scheduler_hints.

* Mix and match resource federation

This is an item that's come up at the past few summits but we finally 
got into some deep discussion about it, the spec is here:

https://review.openstack.org/#/c/342453/

This is about Nova talking to different Cinder vendors/sites. There are 
several problems with Nova managing this though so we agreed that the 
team wanting this would be better served writing a separate proxy 
service which can listen to notifications from nova and cinder and 
maintain a cache/mapping of volumes to the sites they exist in (sort of 
like cells). The proxy service could then also be configured however 
they want, including API policy. The only change to Nova then should be 
configuration only, which is telling Nova which service catalog entry to 
use for volumes, which would point to the proxy. There was an open 
question about how this would work in the cases where Cinder calls back 
to Nova APIs like swap-volume during volume retype/migration, but that's 
only for certain Cinder volume drivers (GlusterFS, NFS, other FS-style 
drivers). George and Kristi were going to investigate that.

* Future of compute host metrics monitoring in Nova

We currently have a CPU metrics monitor in Nova for scheduling. There 
was a blueprint to add memory bandwidth monitoring and another blueprint 
to expose these pluggable monitors and their data from the database via 
the nova-manage CLI. Projects like Watcher use these monitors.

We don't want Nova to be in the business of metrics gathering and 
monitoring, there are several other tools available for doing that. What 
Watcher wants/needs is an ability to influence placement decisions (all 
other things equal between hosts). So the idea came up to provider a 
weight field on resource providers via the placement API so that Watcher 
could use it's time series database (fed from external monitors) to 
adjust weights on resource providers as necessary (keeping in mind 
compute hosts are resource providers too).

So we plan to deprecate the monitor metric plugins as a signal that we 
don't want that monitoring code in nova long-term. Given we don't have a 
solid replacement defined yet, this would be an open-ended deprecation 
as an indication that new services/deployments shouldn't be built on it 
and people should help work on alternatives, like the resource provider 
weight idea.

While on the topic of Watcher (and Joe Cropper was with us), we talked 
about another requirement to be able to pass a list of hosts to the 
migration APIs (live and cold) so that admins don't have to use force 
hosts. There was some agreement in the room that this is generally a 
useful use case / feature, not really only for Watcher. However, there 
was some concern about opening up the API to pass lists of hosts which 
could creep into other APIs and then the Nova API eventually becoming a 
proxy for the placement API. This is an Ocata spec though so more 
discussion will happen in the spec once we open those up.

* Functional devstack (nova-dsvm-integration job)

We could use a dsvm-integration job which has a devstack running but has 
nova in-tree tests that aren't Tempest. This could be useful for testing 
things that rely on libvirt/qemu like the imagebackend refactor series 
related to libvirt storage pools.

Since Sean Dague isn't available to work on this right now we punted.

* Next steps for capabilities/policy exposure in the API

We're going to start small with this in Ocata with a couple of APIs and 
see how things work, like os-attach-interfaces (the Ironic virt driver 
doesn't support that). Then when we're happy with it we can build on it 
for things that don't exist yet, like live resize. We have to keep in 
mind that capabilities are different based on context, i.e. what's 
available in this cloud vs what are you capable (allowed) to do in this 
cloud.

We said this wouldn't be using microversions but would probably be more 
like the metadata API which uses dates.

There were also questions about how to model the capabilities, which 
ties into host capabilities, which is all Ocata summit discussion to be had.

* Live resize

Now that I've mentioned live resize, I'll say we agreed it was something 
we should support (in Ocata) but it's dependent on the 
capabilities/policy exposure in the API. We also said that it would be 
restricted to live resize on the same host and with existing flavors, 
i.e. you can't pass random values like CPU/RAM/Disk for the resize - it 
has to be a flavor. And you can't resize down.

* Moving docs from openstack-manuals into the nova repo

There is a related mailing list thread on this:

http://lists.openstack.org/pipermail/openstack-dev/2016-July/099012.html

There should probably be a cross-project session on this at the summit.

While we understand the issue, we don't think this is something that we 
really want moved to Nova, for a few reasons:

- We don't have anyone in nova that cares enough about docs to own/drive 
this so it's hard to make traction (unlike the api-ref push that Sean 
Dague has done).
- Review time in docs is ~1 day, review time in Nova is ~19 days. So 
moving the docs into Nova actually slows things down for the docs team.
- As an example, Nova doesn't care about writing a SLES install guide.

There was general sentiment that the docs team should feel free to work 
on this for other projects where it works, but doing it in nova will 
probably make things worse.

The TODO on this one was Sean Dague said he'd reply to the above thread 
stating as much.

* Migration state machine

Timofey Durakov is proposing a state machine for orchestration of things 
like live migration which would be heavily tied to the idea of a tasks 
engine. There was discussion about what problem(s) we're actually trying 
to solve (of which there are many) and what concrete steps we can start 
taking now. We agreed that Timofey should start defining what our state 
transitions are right now for various operations like live/cold 
migration and resize. This would be "on paper", i.e. a spec. Then start 
making the code follow the state machine which would probably be new RPC 
calls so we don't try to retrofit the existing code to handle this 
(start clean). Since Ironic people were at the midcycle they also 
pointed out they have are using the automaton library and maybe we could 
follow some of what they are doing.

* Hyper-v cluster driver

In the existing proposal hyper-v changes the instance host underneath 
nova and then the virt driver has to reconcile the changes in the 
resource tracker. This is problematic for a few reasons, but major ones 
are the fact that hyper-v has to anticipate nova's data structures, 
placement logic and the request spec used to build the instance. We also 
plan to eventually move claims from the compute into the scheduler so 
the virt driver wouldn't be able to adjust claims after an out-of-band 
migration. While not great, an alternative would be for hyper-v (the 
hypervisor) to have a nova plugin which would call the Nova REST API and 
tell it where to move an instance.

* Cold migration API improvements

Takashi Natsume has several specs for Ocata to make improvements to the 
cold migration API that exist in the live migration API, like force 
hosts, check destination host and abort support. We agreed that these 
are a good idea since operators would like to fallback to similar 
operations with cold migration if live migration fails.

* The rest

I had to leave after lunch on Thursday to catch my flight home so I 
missed some of the random topics discussed that afternoon. The notes are 
in the etherpad but I'm not sure if there was anything major that came 
up which needs a mention here. If I'm wrong and I just hurt your 
feelings, I'm sorry, please reply and recap any discussion/decisions made.

--

In closing, it was a good time, there was a ton of content and a lot of 
people showed up. I'm a bit scared about how much work we have left to 
do in Newton and we've already started piling on for Ocata, but that's 
the way it is. We definitely have a lot to build on for the upcoming 
summit in October though.

-- 

Thanks,

Matt Riedemann