Open Stack

On 10/01/2018 09:01 AM, Jim Rollenhagen wrote:
> On Mon, Oct 1, 2018 at 8:03 AM Jay Pipes <jaypipes at gmail.com 
> <mailto:jaypipes at gmail.com>> wrote:
> 
>     On 10/01/2018 04:36 AM, John Garbutt wrote:
>      > On Fri, 28 Sep 2018 at 00:46, Jay Pipes <jaypipes at gmail.com
>     <mailto:jaypipes at gmail.com>
>      > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>> wrote:
>      >
>      >     On 09/27/2018 06:23 PM, Matt Riedemann wrote:
>      >      > On 9/27/2018 3:02 PM, Jay Pipes wrote:
>      >      >> A great example of this would be the proposed "deploy
>     template"
>      >     from
>      >      >> [2]. This is nothing more than abusing the placement
>     traits API in
>      >      >> order to allow passthrough of instance configuration data
>     from the
>      >      >> nova flavor extra spec directly into the nodes.instance_info
>      >     field in
>      >      >> the Ironic database. It's a hack that is abusing the entire
>      >     concept of
>      >      >> the placement traits concept, IMHO.
>      >      >>
>      >      >> We should have a way *in Nova* of allowing instance
>     configuration
>      >      >> key/value information to be passed through to the virt
>     driver's
>      >      >> spawn() method, much the same way we provide for
>     user_data that
>      >     gets
>      >      >> exposed after boot to the guest instance via configdrive
>     or the
>      >      >> metadata service API. What this deploy template thing is
>     is just a
>      >      >> hack to get around the fact that nova doesn't have a
>     basic way of
>      >      >> passing through some collated instance configuration
>     key/value
>      >      >> information, which is a darn shame and I'm really kind of
>      >     annoyed with
>      >      >> myself for not noticing this sooner. :(
>      >      >
>      >      > We talked about this in Dublin through right? We said a good
>      >     thing to do
>      >      > would be to have some kind of template/profile/config/whatever
>      >     stored
>      >      > off in glare where schema could be registered on that
>     thing, and
>      >     then
>      >      > you pass a handle (ID reference) to that to nova when
>     creating the
>      >      > (baremetal) server, nova pulls it down from glare and hands it
>      >     off to
>      >      > the virt driver. It's just that no one is doing that work.
>      >
>      >     No, nobody is doing that work.
>      >
>      >     I will if need be if it means not hacking the placement API
>     to serve
>      >     this purpose (for which it wasn't intended).
>      >
>      >
>      > Going back to the point Mark Goddard made, there are two things here:
>      >
>      > 1) Picking the correct resource provider
>      > 2) Telling Ironic to transform the picked node in some way
>      >
>      > Today we allow the use Capabilities for both.
>      >
>      > I am suggesting we move to using Traits only for (1), leaving (2) in
>      > place for now, while we decide what to do (i.e. future of "deploy
>      > template" concept).
>      >
>      > It feels like Ironic's plan to define the "deploy templates" in
>     Ironic
>      > should replace the dependency on Glare for this use case, largely
>      > because the definition of the deploy template (in my mind) is very
>      > heavily related to inspector and driver properties, etc. Mark is
>     looking
>      > at moving that forward at the moment.
> 
>     That won't do anything about the flavor explosion problem, though,
>     right
>     John?
> 
> 
> Does nova still plan to allow passing additional desired traits into the 
> server create request?
> I (we?) was kind of banking on that to solve the Baskin Robbins thing.

That's precisely what I've been looking into. From what I can tell, 
Ironic was planning on using these CUSTOM_DEPLOY_TEMPLATE_XXX traits in 
two ways:

1) To tell Nova what scheduling constraints the instance needed -- e.g. 
"hey Nova, make sure I land on a node that supports UEFI boot mode 
because my boot image relies on that".

2) As a convenient (because it would require no changes to Nova) way of 
passing instance pre-spawn configuration data to the Ironic virt driver 
-- e.g. pass the entire set of traits that are in the RequestSpec's 
flavor and image extra specs to Ironic before calling the Ironic node 
provision API.

#1 is fine IMHO, since it (mostly) represents a "capability" that the 
resource provider (the Ironic baremetal node) must support in order for 
the instance to successfully boot.

#2 is a problem, though, because it *doesn't* represent a capability. In 
fact, it can represent any and all sorts of key/value, JSON/dict or 
other information and this information is not intended to be passed to 
the placement/scheduler service as a constraint. It is this data, also, 
that tends to create the flavor explosion problem because it means that 
Ironic deployers need to create dozens of flavors that vary only 
slightly based on a user's desired deployment configuration.

So, deployers end up needing to create dozens of flavors varying only 
slightly by things like RAID level or some pre-defined disk partitioning 
layout. For example, the deployer might have dozens of flavors that look 
like this:

- 12CPU_128G_RAID10_DRIVE_LAYOUT_X
- 12CPU_128G_RAID5_DRIVE_LAYOUT_X
- 12CPU_128G_RAID01_DRIVE_LAYOUT_X
- 12CPU_128G_RAID10_DRIVE_LAYOUT_Y
- 12CPU_128G_RAID5_DRIVE_LAYOUT_Y
- 12CPU_128G_RAID01_DRIVE_LAYOUT_Y

which is the flavor explosion problem we talk about.

Add into that mix the desire to have certain functionality "activated" 
-- like "this Ironic node supports UEFI boot mode but I need to twiddle 
some foo before I can provision using UEFI boot mode" -- and things just 
get ugly.

I suppose the long-term solution I would like to see is a clean 
separation of #1 from #2, with traits being used as-is for the *required 
capabilities* (i.e. does this node support hardware RAID?) and separate 
command-line arguments for #2, with those command-line arguments being 
references to Glance metadef items. That way, Glance metadefs can be 
used for validating the data supplied to the command line instead of 
Nova needing to add that data validation logic.

So, the long-term command line solution might look something like this:

# list the disk partitioning schemes from glance
openstack metadefs list --object disk_partitioning_scheme

# Choose the disk partitioning scheme you want...
# let's say that this disk partitioning scheme is a JSON
# document that looks like this (the schema for this document
# would belong to the metadef schema in Glance)
{
  "type": "disk_partitioning_scheme",
  "version": 123,
  "object": {
   "raid" : {
    "level": 5
   },
   "partitioning_table_format": "gpt",
   "partitions": {
    "/dev/sda": {
     "format": "ext4",
     "label": "os",
     "size": "256MB",
     "mountpoint": "/"
    },
    "/dev/sdb": {
     "format": "ext4",
     "label": "home",
     "size": "2TB",
     "mountpoint": "/home"
    }...
   }
  },
  "placement": {
   "traits": {
    "required": [
     "STORAGE_DISK_SSD",
     "STORAGE_RAID_HARDWARE",
     "BOOT_MODE_UEFI"
    ]
   }
  }
}

openstack server create --flavor METAL_12CPU_128G \
     --image $IMAGE_UUID \
     --config-data $PARTITIONING_SCHEME_UUID

# nova would see the "placement:traits" collection in the config-data
# object and would merge those trait constraints into the constraints it
# constructs from the flavor extra specs and image metadata, passing
# those constraints directly to placement for scheduling filters.
# Note that this is virtually identical to the solution we developed
# with the Neutron team to put a "network:capabilities" section in the
# port binding profile to indicate the required traits for a port...

# nova would pass the config-data object to the virt driver
# to do with as it wants.

One alternative could also have Ironic folks separate disk partitioning 
from RAID configuration, making separate disk_partitioning_scheme and 
raid_config metadef items in Glance, decoupling the RAID configuration 
from the disk partitioning configuration. The user could then specify 
different RAID config and disk partitioning on the command line, like so:

openstack server create --flavor METAL_12CPU_128G \
     --image $IMAGE_UUID \
     --config-data $PARTIONING_SCHEME_UUID \
     --config-data $RAID_CONFIG_UUID

Anyway, these are the thoughts I'm typing up in the mentioned spec...

-jay