Open Stack

On Thu, Aug 25, 2016 at 04:19:12PM -0400, Zane Bitter wrote:
> On 25/08/16 14:02, Steven Hardy wrote:
> > Hi all,
> > 
> > So I'm following up on a discussion that started here:
> > 
> > https://review.openstack.org/#/c/356240
> > 
> > Basically I recently suggested[1] that relaxing the restriction where a
> > parameter can only exist in exactly one parameter_group would be a way to
> > help work around some pain-points we have in TripleO atm.
> 
> I usually read the scrollback of meetings I'm not able to attend but I must
> have missed that one, sorry.
> 
> > Let me start with the the problems we're trying to solve, because I think
> > they are common to many Heat users, not just TripleO:
> > 
> > 1. When doing nested validation to discover parameter schema, it's
> > impossible to tell if a parameter will be provided by the parent
> 
> IMHO they should all be provided by the parent, but that's another story ;)
> 
> > This is a known issue from when nested validation was first implemented,
> > and we never figured out a satisfactory fix, basically because you can't
> > possibly tell without actually creating things whether an indirectly
> > provided parameter (e.g a reference to another resource in a parent
> > template) will resolve to a valid value.
> > 
> > So when you build a bunch of templates where there are some parameters
> > which form part of an internal interface (e.g they are always provided by
> > the parent and thus should not be exposed to end users) and some which are
> > extra (and should always be provided, or at least exposed to end users) you
> > have no way to differentiate them.
> 
> Heat has a way to differentiate them surely, because it has the templates?
> If you implement it that sounds much more reliable than asking template
> authors to annotate this stuff manually (for a start the same nested
> template can be instantiated in different ways, so it's not even possible in
> general to annotate it in such a way as to indicate which parameters will be
> set by the parent and which should be set by modifying parameter defaults).

The problem is that many (most?) intrinsic functions return None at
validation time, which we can't easily distinguish from no value being
passed into the nested stack.

I'm not saying this can't be fixed, I'm just saying it's probably pretty
hard from an implementation perspective (Jay and I both had a try at fixing
this but I'd welcome some fresh eyes on it!).

Some more context here (and Jay mentions a newton targetted bug although
I'm currently failing to find it):

https://bugs.launchpad.net/heat/+bug/1508857

It's true that no general pattern for annotation is possible, but given a
sufficiently constrained interface (such as we support in some parts of
TripleO) it would be a workable alternative to the status-quo (if the
template annotation was possible, but currently it is not).

> > Example of this here:
> > 
> > https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/heat-api.yaml#L7
> > 
> > ServiceNetMap, DefaultPasswords and EndpointMap are internal interfaces,
> > and the remainder are options for the service that should be exposed to the
> > user.
> > 
> > One option would be to add internal interfaces to a parameter_group
> > "internal", but that then means we can never categorize a parameter as
> > anything else (such as "deprecated" for example, see below).
> > 
> > 2. When you ship a template containing parameters, it's impossible to ever
> > deprecate the parameter names
> > 
> > The problem here is twofold:
> >  - We don't provide any means to tag a parameter as deprecated (so that,
> >    for example, a UI or CLI tool could output an annoying warning to
> >     encourage not using it)
> >  - There's no way to map an old (deprecated) name to a new one unless you
> >    do hacks inside the template such as overwriting one parameter with
> >    another via str_replace (which won't work for all parameter types, and
> >    you still can't ever remove the old parameter because there's no channel
> >    to warn users)
> > 
> > So, one option here is to add a parameter_group called "deprecated" that's
> > then introspected by the client during the validation phase, and outputs a
> > warning when deprecated parameters are used.
> 
> This seems like a very general problem. I would definitely be in favour of
> some native support for this in the HOT format.

Sure, so would I - but I remember discussing this in Vancouver, so it'd be
good to figure out some incremental steps to improve this, again a template
annotation (parameter_groups or something else) provides a simple interim
workaround for this non-trivial and long-standing feature gap.

> > 3. No way to subcategorize more than once
> > 
> > The assumption in the current parameter_group interface is that a UI will
> > always be built on the assumption that parameters should only ever be in
> > one group, which may be true in Horizon, but it's not the only Ux design
> > pattern.
> 
> I guess that's true if you're willing to tweak the layout a bit:
> 
> http://i1.wp.com/www.jeffreythompson.org/blog/wp-content/uploads/2012/07/7-WayVennDiagram-web.png

Haha :)

Now imagine a tree view Ux browsing nested parameters_schema for 100+
openstack services.  You don't always want to group parameters based on the
100+ nested templates, so you need some way to alternately filter them.

> (To be fair a carousel-type UI could actually work quite sanely, with a
> parameter in multiple groups just showing up on multiple different panels.
> But shouldn't we be going for lowest common denominator here? i.e. shouldn't
> the output of validate_template be guaranteed to be usable in all of the UIs
> out there, rather than saying that one UI knows how to display it so
> everything else has to just deal with it?)

Sure, but the lowest common denominator doesn't necessarily have to be
Horizon - the heat integration has (respectfully) always been sparse there,
and in particular support for huge compositions of nested stacks is poor.

With a little more data we can probably enable multiple UI implementations
(incuding Horizon) to improve, so I think we should think carefully before
declaring the status-quo as the lowest common denominator and be done with
it.

> 
> > Particularly when dealing with filtering lots of nested templates (which
> > all accept parameters which may exist in some subcategory, such as
> > "network", "passwords", "advanced" etc.  There's no way to subcategorize
> > parameters in the heat templates, so we're having to wire in hard-coded
> > translations outside of heat, because tripleo-ui doesn't work the same as
> > Horizon (it allows you to browse the nested parameters, and there are a lot
> > so some subcategories are basically needed here, the horizon flat list
> > approach won't work).
> > 
> > Any ideas on the most acceptable path forward here would be appreciated -
> > Randall mentioned enabling per-parameter tags which is certainly an
> > option, and having some means to handle deprecation would also be very
> > good, I'm just not sure on the least impactful way to do this.
> 
> If you made this actually nested, so it output something like:
> 
>     "ParameterGroups": [
>         {
>             "label": "Parameter groups",
>             "description": "My parameter groups",
>             "parameters": [
>                 "param_name-1",
>                 "param_name-2"
>             ],
>             "subgroups": [
>                 {
>                     "label": "Deprecated",
>                     "parameters": [
>                         "param_name-2"
>                     ]
>                 }
>             ]
>         }
>     ]
> 
> then a UI would always display the best approximation possible at the level
> of nesting it chooses to implement (including existing UIs that have already
> 'chosen' no levels of nesting, by virtue of it not being a thing yet).

Interesting, I'd not considered nesting the groups - perhaps that could be
an option, the template level interfaces sounds like they might get a
little unweildy but it's definitely worth some experimenting, thanks!

Steve