Open Stack

Excerpts from Tomas Sedovic's message of 2014-04-04 08:47:46 -0700:
> Hi All,
> 
> I was wondering if the time has come to document what exactly are we
> doing with tripleo-heat-templates and merge.py[1], figure out what needs
> to happen to move away and raise the necessary blueprints on Heat and
> TripleO side.
> 

Yes indeed, it is time.

> (merge.py is a script we use to build the final TripleO Heat templates
> from smaller chunks)
> 
> There probably isn't an immediate need for us to drop merge.py, but its
> existence either indicates deficiencies within Heat or our unfamiliarity
> with some of Heat's features (possibly both).
> 
> I worry that the longer we stay with merge.py the harder it will be to
> move forward. We're still adding new features and fixing bugs in it (at
> a slow pace but still).
>

Merge.py is where we've amassed our debt. We'll pay it back by moving
features into Heat. A huge debt payment is coming in the form of
software config migration, which you mention at the bottom of this
message.

> Below is my understanding of the main marge.py functionality and a rough
> plan of what I think might be a good direction to move to. It is almost
> certainly incomplete -- please do poke holes in this. I'm hoping we'll
> get to a point where everyone's clear on what exactly merge.py does and
> why. We can then document that and raise the appropriate blueprints.
> 
> 
> ## merge.py features ##
> 
> 
> 1. Merging parameters and resources
> 
> Any uniquely-named parameters and resources from multiple templates are
> put together into the final template.
> 
> If a resource of the same name is in multiple templates, an error is
> raised. Unless it's of a whitelisted type (nova server, launch
> configuration, etc.) in which case they're all merged into a single
> resource.
> 
> For example: merge.py overcloud-source.yaml swift-source.yaml
> 
> The final template has all the parameters from both. Moreover, these two
> resources will be joined together:
> 
> #### overcloud-source.yaml ####
> 
>   notCompute0Config:
>     Type: AWS::AutoScaling::LaunchConfiguration
>     Properties:
>       ImageId: '0'
>       InstanceType: '0'
>     Metadata:
>       admin-password: {Ref: AdminPassword}
>       admin-token: {Ref: AdminToken}
>       bootstack:
>         public_interface_ip:
>           Ref: NeutronPublicInterfaceIP
> 
> 
> #### swift-source.yaml ####
> 
>   notCompute0Config:
>     Type: AWS::AutoScaling::LaunchConfiguration
>     Metadata:
>       swift:
>         devices:
>           ...
>         hash: {Ref: SwiftHashSuffix}
>         service-password: {Ref: SwiftPassword}
> 
> 
> The final template will contain:
> 
>   notCompute0Config:
>     Type: AWS::AutoScaling::LaunchConfiguration
>     Properties:
>       ImageId: '0'
>       InstanceType: '0'
>     Metadata:
>       admin-password: {Ref: AdminPassword}
>       admin-token: {Ref: AdminToken}
>       bootstack:
>         public_interface_ip:
>           Ref: NeutronPublicInterfaceIP
>       swift:
>         devices:
>           ...
>         hash: {Ref: SwiftHashSuffix}
>         service-password: {Ref: SwiftPassword}
> 
> 
> We use this to keep the templates more manageable (instead of having one
> huge file) and also to be able to pick the components we want: instead
> of `undercloud-bm-source.yaml` we can pick `undercloud-vm-source` (which
> uses the VirtualPowerManager driver) or `ironic-vm-source`.
> 

The merging of white-listed types is superseded entirely by
OS::Heat::StructuredConfig and OS::Heat::StructuredDeployment. I would
move that we replace all uses of it with those, and deprecate the
feature.

> 
> 
> 2. FileInclude
> 
> If you have a pseudo resource with the type of `FileInclude`, we will
> look at the specified Path and SubKey and put the resulting dictionary in:
> 
> #### overcloud-source.yaml ####
> 
>   NovaCompute0Config:
>     Type: FileInclude
>     Path: nova-compute-instance.yaml
>     SubKey: Resources.NovaCompute0Config
>     Parameters:
>       NeutronNetworkType: "gre"
>       NeutronEnableTunnelling: "True"
> 
> 
> #### nova-compute-instance.yaml ####
> 
>   NovaCompute0Config:
>     Type: AWS::AutoScaling::LaunchConfiguration
>     Properties:
>       InstanceType: '0'
>       ImageId: '0'
>     Metadata:
>       keystone:
>         host: {Ref: KeystoneHost}
>       neutron:
>         host: {Ref: NeutronHost}
>           tenant_network_type: {Ref: NeutronNetworkType}
>           network_vlan_ranges: {Ref: NeutronNetworkVLANRanges}
>           bridge_mappings: {Ref: NeutronBridgeMappings}
>           enable_tunneling: {Ref: NeutronEnableTunnelling}
>           physical_bridge: {Ref: NeutronPhysicalBridge}
>           public_interface: {Ref: NeutronPublicInterface}
>         service-password:
>           Ref: NeutronPassword
>       admin-password: {Ref: AdminPassword}
> 
> The result:
> 
>   NovaCompute0Config:
>     Type: AWS::AutoScaling::LaunchConfiguration
>     Properties:
>       InstanceType: '0'
>       ImageId: '0'
>     Metadata:
>       keystone:
>         host: {Ref: KeystoneHost}
>       neutron:
>         host: {Ref: NeutronHost}
>           tenant_network_type: "gre"
>           network_vlan_ranges: {Ref: NeutronNetworkVLANRanges}
>           bridge_mappings: {Ref: NeutronBridgeMappings}
>           enable_tunneling: "True"
>           physical_bridge: {Ref: NeutronPhysicalBridge}
>           public_interface: {Ref: NeutronPublicInterface}
>         service-password:
>           Ref: NeutronPassword
>       admin-password: {Ref: AdminPassword}
> 
> Note the `NeutronNetworkType` and `NeutronEnableTunneling` parameter
> substitution.
> 
> This is useful when you want to pick only bits and pieces of an existing
> template. In the example above, `nova-compute-instance.yaml` is a
> standalone template you can launch on its own. But when launching the
> overcloud, you don't want to merge nova-compute-instance wholesale. All
> you want is the NovaCompute0Config resource plus a few others.
> 
> I admit I'm not entirely clear on how/why do we need this, though.
> FileInclude is being used just for the overcloud nova compute resources
> (server, config, waitcondition, etc.).
> 
> The undercloud as well as the additional overcloud resources (swift,
> block storage) seem get by without FileInclude.
>

This feature was used so that we could keep nova-compute-instance.yaml
as a whole, working template. I think this use case is also superseded
by the new software config/deployment resources.

> 
> 3. OpenStack::Role metadata key
> 
> I'm not sure what this does or why would we need it. The Ironic
> templates used it but it was removed because they were broken.
> 
> Right now it's used in `tuskar-source.yaml` and `undercloud-source.yaml`
> only.
> 

Servers with the same role will be merged. It is also superseded by
software-config and can probably be removed when we're done with that
migration as wel.

> 
> 4. OpenStack::ImageBuilder::Elements metadata key
> 
> Again, this seems to receive custom handling by merge.py, but I'm
> unclear as to why.
> 

There was a time when I thought we'd drive diskimage-builder using
templates. The long term goal of this was to be able to validate the
metadata based on element content. We have not done any of this, and I
think the feature can just be removed entirely.

> 
> 5. Scaling
> 
> We can mark resources in a Heat template as scalable by giving them the
> '0' suffix. We can then use the `--scale` parameter to make copies of these:
> 
>   SwiftStorage0CompletionHandle:
>     Type: AWS::CloudFormation::WaitConditionHandle
> 
>   SwiftStorage0:
>     Type: OS::Nova::Server
>     Properties:
>       image:
>         {Ref: SwiftStorageImage}
>       flavor: {Ref: OvercloudSwiftStorageFlavor}
>       key_name: {Ref: KeyName}
>     ...
> 
> $ merge.py swift-storage-source.yaml --scale SwiftStorage=2
> 
> result:
> 
>   SwiftStorage0CompletionHandle:
>     Type: AWS::CloudFormation::WaitConditionHandle
> 
>   SwiftStorage0:
>     Type: OS::Nova::Server
>     Properties:
>       image:
>         {Ref: SwiftStorageImage}
>       flavor: {Ref: OvercloudSwiftStorageFlavor}
>       key_name: {Ref: KeyName}
>     ...
> 
>   SwiftStorage1CompletionHandle:
>     Type: AWS::CloudFormation::WaitConditionHandle
> 
>   SwiftStorage1:
>     Type: OS::Nova::Server
>     Properties:
>       image:
>         {Ref: SwiftStorageImage}
>       flavor: {Ref: OvercloudSwiftStorageFlavor}
>       key_name: {Ref: KeyName}
>     ...
> 
> This seems rather close to what OS::Heat::ResourceGroup[2] does. Can we
> just switch to using that instead? I seem to remember reading something
> about the order of deleted resources of being wrong for our purposes.
> 
> Is that correct? What are the specifics?
> 

We have avoided using the heat resource grouping because there's no
opportunity for invididual per-server metadata. We don't want to have
one user/password for the database for all servers, as it creates several
problems, the biggest one being if you change the password for that user
then you have to be sure all operational servers have that password.

With deployments and the random string generator, I think we can achieve
that use case, and make the username/password for each nested stack
available to the database/queue/etc for asserting their availability.

We need to do this not just because merge.py is duplicating functionality
in Heat, but also because this will help us get to Heat managed rolling
updates.

There's also another problem with Heat's groups which is that there is
currently no way to specify which server to scale down from. With the
merge.py method, we can at least generate a template that is too big,
and then manually remove the resources we'd like deleted.

> 
> 6. Merge::Map helper function
> 
> Returns the list of values from a dictionary given as a parameter.
> 
> I'm assuming this would be great to have, but isn't blocking us from
> moving to native Heat templates.
> 
> Clint sent a proposal of this to the Heat developers but it went largely
> unanswered. Perhaps we could revisit it?

I think unanswered is a silent +1, so we should just drive this into
HOT.

> 
> 
> 7-... Whatever I've missed!
> 
> (here's where you get to point out my vast ignorance)


I think you captured all the items I care about.

Ideally we'd eliminate merge.py entirely. Realistically I think we'll
deprecate features one by one and maybe can expect to get rid of
merge.py in a release or two.

> 
> 
> 
> ## a vision of a world without merge.py ##
> 
> Seems to me provider resources ought be able to handle the composability
> side of things (while still allowing us to use them as standalone
> templates) and resource groups should handle the scaling.
> 

> We could keep roughly the same structure: a separate template for each
> OpenStack service (compute, block storage, object storage, ironic, nova
> baremetal). We would then use Heat environments to treat each of these
> templates as a custom resource (e.g. OS::TripleO::Nova,
> OS::TripleO::Swift, etc.).
> 

I've never fully embraced providers for composition. Perhaps I've missed
that as a key feature. An example of this would be helpful. I think if
we deprecated all of merge.py except the "merge unique params and
resources into one template" part, we could probably just start using
nested stacks for that and drop merge.py. However, I'm not a huge fan of
nested stacks as they are a bit clunky. Maybe providers would make that
better?

Anyway, I think I need to see how this would actually work before I can
really grasp it.

> Our overcloud and undercloud templates would then reference all these
> each wrapped in a ResourceGroup or a more sophisticated scaling mechanism.
> 
> 
> 
> ## tangential but nice to have stuff ##
> 
> * move to HOT
>   - It's where any new syntax extensions will happen
> 
> * move to Software Config
>   - Clint has a WIP patch:
>   - https://review.openstack.org/#/c/81666/
> 

When 81666 is done and merged, the next step is to split out the config
and deploy resources into individual services, and then finally we can
switch the syntax to HOT. I am nervous about doing it any faster than
that, as it has already taken quite a while for me to wrap my head
around software configs and the effect it has on the overall deployment.

> 
> What have I missed and what are your thoughts?

I think you covered it nicely, well done.