Open Stack

On 11/23/2015 06:58 AM, Jay Dobies wrote:
> On 11/20/2015 07:05 PM, Ben Nemec wrote:
>> Thinking about this some more makes me wonder if we need a sample config
>> generator like oslo.config.  It would work off something similar to the
>> capabilities map, where you would say
>>
>> SSL:
>>    templates:
>>      -puppet/extraconfig/tls/tls-cert-inject.yaml
>>    output:
>>      -environments/enable-ssl.yaml
>>
>> And the tool would look at that, read all the params from
>> tls-cert-inject.yaml and generate the sample env file.  We'd have to be
>> able to do a few new things with the params in order for this to work:
>>
>> -Need to specify whether a param is intended to be set as a top-level
>> param, parameter_defaults (which we informally do today with the Can be
>> overridden by parameter_defaults comment), or internal, to define params
>> that shouldn't be exposed in the sample config and are only intended as
>> an interface between templates.  There wouldn't be any enforcement of
>> the internal type, but Python relies on convention for its private
>> members so there's precedent. :-)
> 
> There is new functionality in Heat that will let you pass in a series of 
> templates and environments and it will return:
> 
> - The list of top-level parameters, the same way template-validate 
> always did
> - A list of all nested parameters, keyed by resource.
> 
> Take a look at 
> https://github.com/openstack/heat-specs/blob/master/specs/liberty/nested-validation.rst 
> for details and an example.
> 
> That's not entirely what you're getting at, I realize that. I'm glad to 
> see you suggest a convention-based approach because I think that's the 
> only way we're going to be able to convey some of this information.
> 
> I think at the same time we add a mechanism to distinguish between 
> internal and external parameters, we need to add something to indicate 
> required v. optional.
> 
> With a nested stack, anything that's not part of the top-level parameter 
> contract is defaulted. The problem is that it loses information on what 
> is a valid default v. what's simply defaulted to pass validation.

I thought the nested validation spec was supposed to handle that though?
 To me, required vs. optional should be as simple as "Does the parameter
definition have a 'default' key?  If yes, then it's optional, if no,
then it's required for the user to pass a value via a parameter or
parameter_default".  I realize we may not have been following that up to
now for various reasons, but it seems like Heat is already providing a
pretty explicit mechanism for marking params as required, so we ought to
use it.

> 
> I've been noticing this more and more on the vendor integrations. They 
> have parameters that are required (such as a username) and others that 
> are less likely to be changed (I can't think of an example, but I think 
> everyone can see where I'm going with this).
> 
> So I think there are two sorts of things (at least, I'm also thinking 
> off the top of my head) we'd like this tool/sample file to convey:
> 
> - Parameters a user would want to change, as compared to those used for 
> internal data shuffling
> - Information on if the user must supply a value, as compared to 
> parameters with an actual default
> 
> All that said, I dig this idea of a tool that would generate a skeleton 
> environment file.

Cool, I'll try to get a spec written up at some point where we can
bikeshed^W discuss the details.

> 
>> -There would have to be some way to pick out only certain params from a
>> template, since I think there are almost certainly features that are
>> configured using a subset of say puppet/controller.yaml which obviously
>> can't just take the params from an entire file.  Although maybe this is
>> an indication that we could/should refactor the templates to move some
>> of these optional params into their own separate files (at this point I
>> think I should take a moment to mention that this is somewhat of a brain
>> dump, so I haven't thought through all of the implications yet and I'm
>> not sure it all makes sense).
> 
>> The nice thing about generating these programmatically is we would
>> formalize the interface of the templates somewhat, and it would be
>> easier to keep sample envs in sync with the actual implementation.
> 
> You could go so far as to put CI on top of it like we do with the oslo 
> config stuff, which would be neat.
> 
>> You'd never have to worry about someone adding a param to a file but
>> forgetting to update the env (or at least it would be easy to catch and
>> fix when they did, just run "tox -e genconfig").
>>
>> I'm not saying this is a simple or short-term solution, but I'm curious
>> what people think about setting this as a longer-term goal, because as I
>> think our discussion in Tokyo exposed, we're probably going to have a
>> bit of an explosion of sample envs soon and we're going to need some way
>> to keep them sane.
>>
>> Some more comments inline.
>>
>> On 11/19/2015 10:16 AM, Steven Hardy wrote:
>>> On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote:
>>>> On 11/16/2015 04:25 PM, Steven Hardy wrote:
>>>>> Hi all,
>>>>>
>>>>> I wanted to start some discussion re $subject, because it's been apparrent
>>>>> that we have a lack of clarity on this issue (and have done ever since we
>>>>> started using parameter_defaults).
>>>>
>>>> [...]
>>>>
>>>>> How do people feel about this example, and others like it, where we're
>>>>> enabling common, but not mandatory functionality?
>>>>
>>>> At first I was thinking about something as simple as: "don't use top-level
>>>> params for resources which the registry doesn't enable by default".
>>>>
>>>> It seems to be somewhat what we tried to do with the existing pluggable
>>>> resources.
>>>>
>>>> Also, not to hijack the thread but I wanted to add another question related
>>>> to a similar issue:
>>>>
>>>>    Is there a reason to prefer use of parameters: instead of
>>>> parameter_defaults: in the environment files?
>>>>
>>>> It looks to me that by defaulting to parameter_defaults: users won't need to
>>>> update their environment files in case the parameter is moved from top-level
>>>> into a specific nested stack so I'm inclined to prefer this. Are there
>>>> reasons not to?
>>>
>>> The main reason is scope - if you use "parameters", you know the data flow
>>> happens via the parent template (e.g overcloud-without-mergepy) and you
>>> never have to worry about naming collisions outside of that template.
>>>
>>> But if you use parameter_defaults, all parameters values defined that way
>>> are effectively global, and you then have to be much more careful that you
>>> never shadow a parameter name and get an unexpected value passed in to it.
>>>
>>> Here's another example of why we need to decide this btw:
>>>
>>> https://review.openstack.org/#/c/229471/
>>>
>>> Here, we have some workers parameters, going only into controller.yaml -
>>> this is fine, but the new options are completely invisible to users who
>>> look at the overcloud_without_mergepy parameters schema as their interface
>>> (in particular I'm thinking of any UI here).
>>>
>>> My personal preference is to say:
>>>
>>> 1. Any templates which are included in the default environment (e.g
>>> overcloud-resource-registry-puppet.yaml), must expose their parameters
>>> via overcloud-without-mergepy.yaml
>>>
>>> 2. Any templates which are included in the default environment, but via a
>>> "noop" implementation *may* expose their parameters provided they are
>>> common and not implementation/vendor specific.
>>
>> This seems like a reasonable approach, although that "may" still leaves
>> a lot of room for bikeshedding. ;-)
>>
>> It might be good to say that in this case it is "preferred" to use a
>> top-level param, but if there's a reason not to then it's acceptable to
>> use parameter_defaults.  An example for the SSL case would be the
>> certificate path - I specifically do not want that visibly exposed to
>> the user at this point, so I wouldn't want it added to the top-level
>> template.  I consider that an implementation detail where if you know
>> what you're doing you can override it, but otherwise you shouldn't touch it.
>>
>>>
>>> 3. Any templates exposing vendor specific interfaces (e.g at least anything
>>> related to the OS::TripleO::*ExtraConfig* interfaces) must not expose any
>>> parameters via the top level template.
>>>
>>> How does this sound?
>>>
>>> This does mean we suffer some template bloat from (1) and (2), but it makes
>>> the job of any UI or other tool requiring user input much easier, I think?
>>>
>>> Steve
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>