Open Stack

On 11/27/2013 10:24 AM, Tim Schnell wrote:
> So the originally question that I attempted to pose was, "Can we add a
> schema-less metadata section to the template that can be used for a
> variety of purposes?". It looks like the answer is no, we need to discuss
> the features that would go in the metadata section and add them to the HOT
> specification if they are viable. I don't necessarily agree with this
> answer but I accept it as viable and take responsibility for the
> long-winded process that it took to get to this point.
>
> I think some valid points have been made and I have re-focused my efforts
> into the following proposed solution.
>
> I am fine with getting rid of the concept of a schema-less metadata
> section. If we can arrive at a workable design for a few use cases then I
> think that we won't need to discuss any of the options that Zane mentioned
> for handling the metadata section, comments, separate file, or in the
> template body.
>
> Use Case #1
> I see valid value in being able to group templates based on a type or
> keyword. This would allow any client, Horizon or a Template Catalog
> service, to better organize and handle display options for an end-user.
>
> I believe that Ladislav initially proposed a solution that will work here.
> So I will second a proposal that we add a new top-level field to the HOT
> specification called "keywords" that contains this template type.
>
> 	keywords: wordpress, mysql, etcŠ
+1, but lets make the most of yaml and give it structure:
keywords: [wordpress, mysql, lamp]
keywords:
- wordpress
- mysql
- lamp

>
> Use Case #2
> The template author should also be able to explicitly define a help string
> that is distinct and separate from the description of an individual
> parameter. An example where this use case originated was with Nova
> Keypairs. The description of a keypair parameter might be something like,
> "This is the name of a nova key pair that will be used to ssh to the
> compute instance." A help string for this same parameter would be, "To
> learn more about nova keypairs click on this help article."
>
> I propose adding an additional field to the parameter definition:
> 	
> 	Parameters:
> 		<parameter name>:
> 			description: This is the name of a nova key pair that will be used to
> ssh to the compute instance.
> 			help: To learn more about nova key pairs click on this <a
> href="/some/url/">help article</a>.
+1, this could be used today in the stack create screens of horizon.

> Use Case #3
> Grouping parameters would help the client make smarter decisions about how
> to display the parameters for input to the end-user. This is so that all
> parameters related to some database resource can be intelligently grouped
> together. In addition to grouping these parameters together, there should
> be a method to ensuring that the order within the group of parameters can
> be explicitly stated. This way, the client can return a group of database
> parameters and the template author can indicate that the database instance
> name should be first, then the username, then the password, instead of
> that group being returned in a random order.
>
> 	Parameters:
> 		db_name:
> 			group: db
> 			order: 0
> 		db_username:
> 			group: db
> 			order: 1
> 		db_password:
> 			group: db
> 			order: 2
> 		web_node_name:
> 			group: web_node
> 			order: 0
> 		keypair:
> 			group: web_node
> 			order: 1
>
> These are the use cases that have been clearly defined. The original
> purpose of the metadata section was to "future-proof" (I say future-proof,
> you say pre-optimize ;) ) rapid iterations to potential client design. The
> intent of the strategy was so that we did not overburden the Heat Core
> team with requirements that may fluctuate or change as we attempt to
> improve the user experience of Heat for the community. In hindsight I can
> see how that intent was misconstrued and may have come off as
> condescending and I apologize for that.
We *need* a solution for grouping and ordering in horizon for generating
the parameter fields. All that is being done currently (I think) is
sorting by mandatory, then parameter name, which often pushes important
parameters to the bottom of the screen. I'd prefer explicit attributes
rather than inferring from a special parser that preserves the order of
parameters in the template. The parameter fields are built from the
results of a template-validate call.

Also, a group should really have its own order and description to
display, so we may need a new top-level section, eg
parameter-groups:
  db:
    description: Database configuration options
    order: 1
  web_node:
    description: Web server configuration
    order: 0
parameters:
  # as above

A template-validate call would need to return all the group and ordering
information, but otherwise heat can ignore this extra data.
> I think that these proposed use cases will add real value to Heat and
> would require only changes to the HOT specification. If these are
> acceptable please let me know and I will drop the existing blueprint and
> patches in favor of something that implements this design. If there are
> issues to discuss here please suggest alternative solutions so that we can
> continue to move forward with these improvements.
I think there are 3 potential blueprints here which all have merit. I
struck use cases #2 and #3 when I was working on heat horizon, so these
would have immediate benefit to a real open-source heat UI.
> I have no doubt that this is just the beginning of plenty of requirements
> being driven from an attempt to improve the overall end user experience of
> Heat and I will continue to vet the use cases and proposed implementations
> with the community before suggesting a solution. I know that this proposal
> leaves out a few use cases that have been discussed like template
> author/version and that is because I believe that some of the arguments
> made are valid and should be looked into before proposing changes to Heat.
> Most of those other use cases probably can be served from some other
> solution like git or the future template catalog.
>
> Thanks,	
> Tim
>
>
>
>
>
> On 11/26/13 12:31 PM, "Zane Bitter" <zbitter at redhat.com> wrote:
>
>> On 26/11/13 03:26, Keith Bray wrote:
>>>
>>> On 11/25/13 5:46 PM, "Clint Byrum" <clint at fewbar.com> wrote:
>>>
>>>> Excerpts from Tim Schnell's message of 2013-11-25 14:51:39 -0800:
>>>>> Hi Steve,
>>>>>
>>>>> As one of the UI developers driving the requirements behind these new
>>>>> blueprints I wanted to take a moment to assure you and the rest of the
>>>>> Openstack community that the primary purpose of pushing these
>>>>> requirements
>>>>> out to the community is to help improve the User Experience for Heat
>>>>> for
>>>>> everyone. Every major UI feature that I have implemented for Heat has
>>>>> been
>>>>> included in Horizon, see the Heat Topology, and these requirements
>>>>> should
>>>>> improve the value of Heat, regardless of the UI.
>>>>>
>>>>>
>>>>> Stack/template metadata
>>>>> We have a fundamental need to have the ability to reference some
>>>>> additional metadata about a template that Heat does not care about.
>>>>> There
>>>>> are many possible use cases for this need but the primary point is
>>>>> that
>>>>> we
>>>>> need a place in the template where we can iterate on the schema of the
>>>>> metadata without going through a lengthy design review. As far as I
>>>>> know,
>>>>> we are the only team attempting to actually productize Heat at the
>>>>> moment
>>>>> and this means that we are encountering requirements and requests that
>>>>> do
>>>>> not affect Heat directly but simply require Heat to allow a little
>>>>> wiggle
>>>>> room to flesh out a great user experience.
>>>>>
>>>> Wiggle room is indeed provided. But reviewers need to understand your
>>>> motivations, which is usually what blueprints are used for. If you're
>>>> getting push back, it is likely because your blueprints to not make the
>>>> use cases and long term vision obvious.
>>> Clint, can you be more specific on what is not clear about the use case?
>> The part where it's used for something. What I'm hearing in this thread
>> is "We need this for a template catalog, but we're not going to talk
>> about that until later. Trust us, though, we really need it."
>>
>> That's a problem, because none of this stuff makes sense except in the
>> context of the template catalog. What does the template catalog do? Who
>> writes the templates in the catalog? Why is the template catalog not
>> backed by git? Who can say which parts of the solution belong in the
>> template catalog and which in Heat when we don't even know what it's a
>> solution to?
>>
>>> What I am seeing is that the use case of meta data is not what is being
>> I'm contesting it:
>>
>> https://review.openstack.org/#/c/56703/7/doc/source/template_guide/hot_spe
>> c.rst
>>
>> The problem here isn't that the process wasn't followed, it's that all
>> of these things - one-shot outputs, operator-specific structured
>> metadata sections in templates, APIs to report versions of randomly
>> selected daemons - are fundamentally bad ideas. And they're all bad in
>> exactly the same way: they demand that Heat implement something that it
>> can only do in a half-baked manner, because it's slightly easier (or
>> seems slightly easier) than implementing something in some external tool
>> that can work.
>>
>> It's difficult to escape the conclusion that these features have been
>> decided upon behind closed doors with the community excluded, by people
>> who are not especially familiar with Heat, with a view to minimising the
>> effort required to implement some other (proprietary?) thing and not
>> necessarily to coming up with the best implementation. If this is your
>> development process then you are Doing Open Source Wrong. At a very
>> minimum you need to work about 10x harder to explain all of these design
>> decisions - in particular the reasons why the obvious alternatives that
>> don't appear to have been considered are insufficient - to the
>> community. It's much, much easier, though, to just do your design in the
>> open with community involvement.
>>
>> Take this metadata section idea. (Please!) Let's step back and examine
>> the very last decision in the chain - assume for a moment that we accept
>> that this metadata needs to be included in the template. There are three
>> obvious ways to do that:
>>
>>  1) Include them as YAML comments
>>  2) Include them in the body of the template
>>  3) Include them as a separate YAML document in the same file
>>
>> How many of those were actually considered? My bet is only #2. It
>> doesn't really matter though, because there is not one word of design
>> documentation that would help me understand the reasons for rejecting #1
>> & #3 (neither of which require any changes to Heat or the template
>> format). How can I have confidence that #2 is the correct solution to
>> the problem? Especially if I don't agree (I don't)?
>>
>> The number of options not considered grows (and my confidence in the
>> solution shrinks) exponentially as you step all the way back through the
>> chain of undocumented decisions to the actual thing that drove this idea.
>>
>>> contested, but that the Blueprint of where meta data should go is being
>>> contested by only a few (but not all) of the core devs.  The Blueprint
>>> for
>> I contested that first because without knowing what the feature is even
>> for, there's nothing else to contest.
>>
>>> in-template metadata was already approved for Icehouse, but now that
>>> work
>>> has been delivered on the implementation of that blueprint, the
>>> blueprint
>>> itself is being contested:
>>>     https://blueprints.launchpad.net/heat/+spec/namespace-stack-metadata
>> It seems clear that we have to communicate better to new contributors
>> what the expectations are around getting features in. But I would be
>> really sad if we have to jump straight in to the sort of legalistic
>> processes that some of the larger OpenStack programs follow. This is
>> still a relatively small project; we should be able to handle these
>> questions by talking to each other.
>>
>>> I'd like to propose that the blueprint that has been accepted go forth
>>> with the code that exactly implements it, and if there are alternative
>>> proposals and appropriate reasons for the community to come to consensus
>>> on a different approach, that we then iterate and move the data
>>> (deprecate
>>> the older feature if necessary, e.g. If that decision comes after
>>> Icehouse, else of a different/better implementation comes before
>>> Icehouse,
>>> then no harm done).
>> -1
>>
>>>>> There is precedence for an optional metadata section that can contain
>>>>> any
>> As Clint said, you're comparing APIs to templates. Those are very
>> different.
>>
>>>>> end-user data in other Openstack projects and it is necessary in order
>>>>> to
>>>>> iterate quickly and provide value to Heat.
>> Non sequitur.
>>
>>>> Nobody has said you can't have meta-data on stacks, which is what other
>>>> projects use.
>>>>
>>>>> There are many use cases that can be discussed here, but I wanted to
>>>>> reiterate an initial discussion point that, by definition,
>>>>> "stack/template_metadata" does not have any hard requirements in terms
>>>>> of
>>>>> schema or what does or does not belong in it.
>>>>>
>>>>> One of the initial use cases is to allow template authors to
>>>>> categorize
>>>>> the template as a specific "type".
>>>>>
>>>>>      template_metadata:
>>>>>          short_description: Wordpress
>> First of all, "short_description" is a dreadful name for something that
>> is basically tags.
>>
>> Secondly, these are tags in the template catalog. So why are they
>> incorporated in the template, and not stored in the template catalog?
>>
>> Finally, if you really want this it can be trivially implemented using
>> comments.
>>
>>>> Interesting. Would you support adding a "category" keyword to python so
>>>> we don't have to put it in setup.cfg and so that the egg format doesn't
>>>> need that section? Pypi can just parse the python to categorize the
>>>> apps
>>>> when they're uploaded. We could also have a file on disk for qcow2
>>>> images
>>>> that we upload to glance that will define the meta-data.
>>>>
>>>> To be more direct, I don't think the templates themselves are where
>>>> this
>>>> meta-data belongs. A template is self-aware by definition, it doesn't
>>>> need the global metadata section to tell it that it is WordPress. For
>>>> anything else that needs to be globally referenced there are
>>>> parameters.
>>>> Having less defined inside the template means that you get _more_
>>>> wiggle
>>>> room for your template repository.
>>> Clint, you are correct that the Template does not need to know what it
>>> is.
>>>   It's every other service (and users of those services) that a Template
>>> passes through or to that would care to know what it is. We are
>>> suggesting
>>> we put that meta data in the template file and expressly ignore it for
>>> purposes of parsing the template language in the Heat engine, so we
>>> agree
>>> it not a necessary part of the template.  Sure, we could encode the
>>> metadata info in a separate catalog...  but, take the template out of
>>> the
>>> catalog and now all that useful associated data is lost or would need to
>>> be recreated by someone or some service.  That does not make the
>>> template
>>> portable, and that is a key aspect of what we are trying to achieve (all
>>> user-facing clients, like Horizon, or humans reading the file, can take
>>> advantage). We don't entirely know yet what is most useful in
>>> portability
>> What's most useful in portability is placing the minimum requirements
>> (especially undocumented and/or non-standard requirements) on the
>> template author, since you can't control the template author.
>>
>>> and what isn't, so meta data in-template provides the "wiggle room"
>>> innovation space to suss that out.  We already know of some specific use
>>> cases of data that we feel are important, which Tim identified one
>>> specific example.. As specific metadata items become popular or prove to
>>> be useful to rely on by the larger community or service operators
>>> (public
>>> and private) of Heat, we as a community can drive that information back
>>> into the schema for the template or some portable format mechanism.
>>>
>>>> I 100% support having a template catalog. IMO it should be glance,
>>>> which is our catalog service in OpenStack. Who cares if nova or heat
>>>> are
>>>> consuming images or templates. It is just sharable blobs of data and
>>>> meta-data in a highly scalable service. It already has the concept of
>>>> global and tenant-scope. It just needs an image type of 'hot' and then
>>>> heat can start consuming templates from glance. And the template
>>>> authors
>>>> should maintain some packaging meta-data in glance to communicate to
>>>> users that this is "Wordpress" and "Single-Node". If Glance's meta-data
>>>> is too limiting, expand it! I'm sure image authors and consumers would
>>>> appreciate that.
>>> This is definitely interesting... And takes the long view IMO.  Let me
>>> explain:  I don't anticipate Heat catalog'ing in Glance is something
>>> that
>>> has a high chance of getting implemented in the Icehouse timeframe (at
>>> least, not more so than in-template metadata), do you?  From a SOA
>>> service
>>> deployer perspective, I'm sure you can appreciate that rolling out new
>>> functionality in Glance to support an Orchestration project use case is
>>> not simple, and requires strong business justification and coordination
>>> as
>>> an operator of a cloud.. One worth exploring for sure, but not the go-to
>>> default strategy. I view the metadata change as very minor with little
>>> to
>>> no disruption to any service, including Heat (Heat just ignores the
>> I view it as an open invitation to fragment the template format and the
>> ecosystem.
>>
>>> metadata, completely).. This fits very well in an iterative development
>>> model.  New blueprints could be raised, as you suggested, to move
>>> metadata
>>> and catalog features into Glance.  My concern is that if we go the
>>> Glance
>>> route now, we are encouraging a precedent that we aren't iterative (due
>>> to
>>> abandoning an already accepted blueprint in favor of a more complex and
>>> time intensive solution) and that we won't get this implemented within
>>> the
>>> current release cycle.
>>>
>>>>> This would let the client of the Heat API group the templates by type
>>>>> which would create a better user experience when selecting or managing
>>>>> templates. The the end-user could select "Wordpress" and drill down
>>>>> further to select templates with different options, "single node", "2
>>>>> web
>>>>> nodes", etc...
>>>>>
>>>> That is all api stuff, not language stuff.
>>> If this were done solely at the API, it would have to be maintained
>>> 1-to-1
>>> with a template (in which case there is an implicit and explicit
>>> association), and exported with the template in order to port the
>>> template.
>> I don't understand all this stuff about 'portability'. From your
>> previous descriptions of the template catalog, it sounded like it's for
>> templates hand-picked by the operator. Why would one operator want to
>> import the tags defined by another operator anyway? And Why wouldn't
>> that just be a git clone of the backing repository? And how does any of
>> this make the template more "portable"???
>>
>>>>> Once a feature has consistently proven that it adds value to Heat or
>>>>> Horizon, then I would suggest that we can discuss the schema for that
>>>>> feature and codify it then.
>>>>>
>>>>> In order to keep the discussion simple, I am only responding to the
>>>>> need
>>>>> for stack/template metadata at the moment but I'm sure discussions on
>>>>> the
>>>>> management api and template catalog will follow.
>>>>>
>>>> Your example puts the template catalog in front of this feature, and I
>>>> think that exposes this feature as misguided.
>>> I'd like template author and creation date portable so that it goes
>>> wherever the template goes in a portable format that consumers (any
>>> client) of the Heat service can understand.  I'd also like this
>> Put it in a comment.
>>
>>> information to be available for a deployed stack in Heat, so it is need
>>> not be catalog specific.  If author/creation-date information were only
>> If it needs to be available for a deployed stack (you haven't made an
>> argument for this yet) then add it into the template-proper. If Heat
>> needs to read it then some sort of wild-west schemaless metadata area is
>> definitely not the place to put it.
>>
>> Alternately, maybe we need to look at storing templates in such a way
>> that the comments are preserved. I would definitely look favourably on a
>> blueprint to that effect.
>>
>>> available in the catalog, then we would have to entirely wrap the Heat
>>> API
>>> in another service or risk drift in the data between Heat and the
>>> separate
>>> non-wrapping catalog service. Instead of raising a blueprint for every
>>> piece of data that may be useful to deploying this service (some of
>>> which
>> Please don't raise a blueprint for every patch. e.g. a feature and its
>> documentation 
>> (https://blueprints.launchpad.net/heat/+spec/update-stack-metadata-spec)
>> are not two blueprints.
>>
>>> certain folks may not care about [e.g. Many of the folks arguing against
>>> metadata all together], the in-template metadata is, IMO, a suitable
>>> approach to experiment and then we can drive back experiential use-cases
>>> into future improvements (e.g. in-language schema changes or catalog
>>> changes).  At Rackspace we have experience running a nearly identical
>>> service, so we know this data, in this particular place, is a valid use
>>> case for consumers of an Orchestration service.
>>> Example:
>>> (1) Cloud customer of service operator deploys a template (may or may
>>> not
>>> have come from the catalog)
>> If it didn't come from the catalog provided by the operator, what are
>> the chances that it contains this information anyway? How is the average
>> template author going to know about the schema for the metadata read by
>> your proprietary tool?
>>
>>> (2) Customer encounters problem with the stack and calls support
>>> (3) Support specialist [not the service operator/developer] is trying to
>>> figure out who to contact for additional help given failures seen with
>>> the
>>> stack. The dev team of the service is not the expert in application foo
>>> and did not develop the template. It's not a Heat issue, it's a
>>> template/application foo issue.
>> This sounds like a good argument for Heat preserving comments.
>>
>>> Having this detail come from Glance, as you pointed out, would be great.
>>> But, my hope is that many folks will develop and share templates, and
>>> the
>>> folks developing templates won't necessarily be running glance or have
>>> access to upload their own images into glance (that is a service
>>> operator
>>> choice).  If we force this basic info to glance, we are limiting our
>>> portability (and therefore adoption story) of Heat.
>> What's really going to limit portability is having multiple overlapping
>> undocumented proprietary schemas for parts of the template. I actually
>> think this has the potential to damage the Heat ecosystem, and that's
>> one of the reasons I'm pushing back on it.
>>
>> regards,
>> Zane.
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev