Open Stack

On 11/26/13 4:16 PM, "Steve Baker" <sbaker at redhat.com> wrote:

>On 11/27/2013 10:24 AM, Tim Schnell wrote:
>> So the originally question that I attempted to pose was, "Can we add a
>> schema-less metadata section to the template that can be used for a
>> variety of purposes?". It looks like the answer is no, we need to
>>discuss
>> the features that would go in the metadata section and add them to the
>>HOT
>> specification if they are viable. I don't necessarily agree with this
>> answer but I accept it as viable and take responsibility for the
>> long-winded process that it took to get to this point.
>>
>> I think some valid points have been made and I have re-focused my
>>efforts
>> into the following proposed solution.
>>
>> I am fine with getting rid of the concept of a schema-less metadata
>> section. If we can arrive at a workable design for a few use cases then
>>I
>> think that we won't need to discuss any of the options that Zane
>>mentioned
>> for handling the metadata section, comments, separate file, or in the
>> template body.
>>
>> Use Case #1
>> I see valid value in being able to group templates based on a type or
>> keyword. This would allow any client, Horizon or a Template Catalog
>> service, to better organize and handle display options for an end-user.
>>
>> I believe that Ladislav initially proposed a solution that will work
>>here.
>> So I will second a proposal that we add a new top-level field to the HOT
>> specification called "keywords" that contains this template type.
>>
>> 	keywords: wordpress, mysql, etcŠ
>+1, but lets make the most of yaml and give it structure:
>keywords: [wordpress, mysql, lamp]
>keywords:
>- wordpress
>- mysql
>- lamp
>
>>
>> Use Case #2
>> The template author should also be able to explicitly define a help
>>string
>> that is distinct and separate from the description of an individual
>> parameter. An example where this use case originated was with Nova
>> Keypairs. The description of a keypair parameter might be something
>>like,
>> "This is the name of a nova key pair that will be used to ssh to the
>> compute instance." A help string for this same parameter would be, "To
>> learn more about nova keypairs click on this help article."
>>
>> I propose adding an additional field to the parameter definition:
>> 	
>> 	Parameters:
>> 		<parameter name>:
>> 			description: This is the name of a nova key pair that will be used to
>> ssh to the compute instance.
>> 			help: To learn more about nova key pairs click on this <a
>> href="/some/url/">help article</a>.
>+1, this could be used today in the stack create screens of horizon.
>
>> Use Case #3
>> Grouping parameters would help the client make smarter decisions about
>>how
>> to display the parameters for input to the end-user. This is so that all
>> parameters related to some database resource can be intelligently
>>grouped
>> together. In addition to grouping these parameters together, there
>>should
>> be a method to ensuring that the order within the group of parameters
>>can
>> be explicitly stated. This way, the client can return a group of
>>database
>> parameters and the template author can indicate that the database
>>instance
>> name should be first, then the username, then the password, instead of
>> that group being returned in a random order.
>>
>> 	Parameters:
>> 		db_name:
>> 			group: db
>> 			order: 0
>> 		db_username:
>> 			group: db
>> 			order: 1
>> 		db_password:
>> 			group: db
>> 			order: 2
>> 		web_node_name:
>> 			group: web_node
>> 			order: 0
>> 		keypair:
>> 			group: web_node
>> 			order: 1
>>
>> These are the use cases that have been clearly defined. The original
>> purpose of the metadata section was to "future-proof" (I say
>>future-proof,
>> you say pre-optimize ;) ) rapid iterations to potential client design.
>>The
>> intent of the strategy was so that we did not overburden the Heat Core
>> team with requirements that may fluctuate or change as we attempt to
>> improve the user experience of Heat for the community. In hindsight I
>>can
>> see how that intent was misconstrued and may have come off as
>> condescending and I apologize for that.
>We *need* a solution for grouping and ordering in horizon for generating
>the parameter fields. All that is being done currently (I think) is
>sorting by mandatory, then parameter name, which often pushes important
>parameters to the bottom of the screen. I'd prefer explicit attributes
>rather than inferring from a special parser that preserves the order of
>parameters in the template. The parameter fields are built from the
>results of a template-validate call.
>
>Also, a group should really have its own order and description to
>display, so we may need a new top-level section, eg
>parameter-groups:
>  db:
>    description: Database configuration options
>    order: 1
>  web_node:
>    description: Web server configuration
>    order: 0
>parameters:
>  # as above
>
>A template-validate call would need to return all the group and ordering
>information, but otherwise heat can ignore this extra data.

I agree with all of your modifications, although bringing up the
template-validate call reminded me that the implementation of this use
case should also imply a new REST endpoint specifically for returning
parameters. It seems like the current implementation in Horizon is a bit
hack-y by calling template-validate instead of something like
"get-parameters".

>> I think that these proposed use cases will add real value to Heat and
>> would require only changes to the HOT specification. If these are
>> acceptable please let me know and I will drop the existing blueprint and
>> patches in favor of something that implements this design. If there are
>> issues to discuss here please suggest alternative solutions so that we
>>can
>> continue to move forward with these improvements.
>I think there are 3 potential blueprints here which all have merit. I
>struck use cases #2 and #3 when I was working on heat horizon, so these
>would have immediate benefit to a real open-source heat UI.
>> I have no doubt that this is just the beginning of plenty of
>>requirements
>> being driven from an attempt to improve the overall end user experience
>>of
>> Heat and I will continue to vet the use cases and proposed
>>implementations
>> with the community before suggesting a solution. I know that this
>>proposal
>> leaves out a few use cases that have been discussed like template
>> author/version and that is because I believe that some of the arguments
>> made are valid and should be looked into before proposing changes to
>>Heat.
>> Most of those other use cases probably can be served from some other
>> solution like git or the future template catalog.
>>
>> Thanks,	
>> Tim
>>
>>
>>
>>
>>
>> On 11/26/13 12:31 PM, "Zane Bitter" <zbitter at redhat.com> wrote:
>>
>>> On 26/11/13 03:26, Keith Bray wrote:
>>>>
>>>> On 11/25/13 5:46 PM, "Clint Byrum" <clint at fewbar.com> wrote:
>>>>
>>>>> Excerpts from Tim Schnell's message of 2013-11-25 14:51:39 -0800:
>>>>>> Hi Steve,
>>>>>>
>>>>>> As one of the UI developers driving the requirements behind these
>>>>>>new
>>>>>> blueprints I wanted to take a moment to assure you and the rest of
>>>>>>the
>>>>>> Openstack community that the primary purpose of pushing these
>>>>>> requirements
>>>>>> out to the community is to help improve the User Experience for Heat
>>>>>> for
>>>>>> everyone. Every major UI feature that I have implemented for Heat
>>>>>>has
>>>>>> been
>>>>>> included in Horizon, see the Heat Topology, and these requirements
>>>>>> should
>>>>>> improve the value of Heat, regardless of the UI.
>>>>>>
>>>>>>
>>>>>> Stack/template metadata
>>>>>> We have a fundamental need to have the ability to reference some
>>>>>> additional metadata about a template that Heat does not care about.
>>>>>> There
>>>>>> are many possible use cases for this need but the primary point is
>>>>>> that
>>>>>> we
>>>>>> need a place in the template where we can iterate on the schema of
>>>>>>the
>>>>>> metadata without going through a lengthy design review. As far as I
>>>>>> know,
>>>>>> we are the only team attempting to actually productize Heat at the
>>>>>> moment
>>>>>> and this means that we are encountering requirements and requests
>>>>>>that
>>>>>> do
>>>>>> not affect Heat directly but simply require Heat to allow a little
>>>>>> wiggle
>>>>>> room to flesh out a great user experience.
>>>>>>
>>>>> Wiggle room is indeed provided. But reviewers need to understand your
>>>>> motivations, which is usually what blueprints are used for. If you're
>>>>> getting push back, it is likely because your blueprints to not make
>>>>>the
>>>>> use cases and long term vision obvious.
>>>> Clint, can you be more specific on what is not clear about the use
>>>>case?
>>> The part where it's used for something. What I'm hearing in this thread
>>> is "We need this for a template catalog, but we're not going to talk
>>> about that until later. Trust us, though, we really need it."
>>>
>>> That's a problem, because none of this stuff makes sense except in the
>>> context of the template catalog. What does the template catalog do? Who
>>> writes the templates in the catalog? Why is the template catalog not
>>> backed by git? Who can say which parts of the solution belong in the
>>> template catalog and which in Heat when we don't even know what it's a
>>> solution to?
>>>
>>>> What I am seeing is that the use case of meta data is not what is
>>>>being
>>> I'm contesting it:
>>>
>>> 
>>>https://review.openstack.org/#/c/56703/7/doc/source/template_guide/hot_s
>>>pe
>>> c.rst
>>>
>>> The problem here isn't that the process wasn't followed, it's that all
>>> of these things - one-shot outputs, operator-specific structured
>>> metadata sections in templates, APIs to report versions of randomly
>>> selected daemons - are fundamentally bad ideas. And they're all bad in
>>> exactly the same way: they demand that Heat implement something that it
>>> can only do in a half-baked manner, because it's slightly easier (or
>>> seems slightly easier) than implementing something in some external
>>>tool
>>> that can work.
>>>
>>> It's difficult to escape the conclusion that these features have been
>>> decided upon behind closed doors with the community excluded, by people
>>> who are not especially familiar with Heat, with a view to minimising
>>>the
>>> effort required to implement some other (proprietary?) thing and not
>>> necessarily to coming up with the best implementation. If this is your
>>> development process then you are Doing Open Source Wrong. At a very
>>> minimum you need to work about 10x harder to explain all of these
>>>design
>>> decisions - in particular the reasons why the obvious alternatives that
>>> don't appear to have been considered are insufficient - to the
>>> community. It's much, much easier, though, to just do your design in
>>>the
>>> open with community involvement.
>>>
>>> Take this metadata section idea. (Please!) Let's step back and examine
>>> the very last decision in the chain - assume for a moment that we
>>>accept
>>> that this metadata needs to be included in the template. There are
>>>three
>>> obvious ways to do that:
>>>
>>>  1) Include them as YAML comments
>>>  2) Include them in the body of the template
>>>  3) Include them as a separate YAML document in the same file
>>>
>>> How many of those were actually considered? My bet is only #2. It
>>> doesn't really matter though, because there is not one word of design
>>> documentation that would help me understand the reasons for rejecting
>>>#1
>>> & #3 (neither of which require any changes to Heat or the template
>>> format). How can I have confidence that #2 is the correct solution to
>>> the problem? Especially if I don't agree (I don't)?
>>>
>>> The number of options not considered grows (and my confidence in the
>>> solution shrinks) exponentially as you step all the way back through
>>>the
>>> chain of undocumented decisions to the actual thing that drove this
>>>idea.
>>>
>>>> contested, but that the Blueprint of where meta data should go is
>>>>being
>>>> contested by only a few (but not all) of the core devs.  The Blueprint
>>>> for
>>> I contested that first because without knowing what the feature is even
>>> for, there's nothing else to contest.
>>>
>>>> in-template metadata was already approved for Icehouse, but now that
>>>> work
>>>> has been delivered on the implementation of that blueprint, the
>>>> blueprint
>>>> itself is being contested:
>>>>     
>>>>https://blueprints.launchpad.net/heat/+spec/namespace-stack-metadata
>>> It seems clear that we have to communicate better to new contributors
>>> what the expectations are around getting features in. But I would be
>>> really sad if we have to jump straight in to the sort of legalistic
>>> processes that some of the larger OpenStack programs follow. This is
>>> still a relatively small project; we should be able to handle these
>>> questions by talking to each other.
>>>
>>>> I'd like to propose that the blueprint that has been accepted go forth
>>>> with the code that exactly implements it, and if there are alternative
>>>> proposals and appropriate reasons for the community to come to
>>>>consensus
>>>> on a different approach, that we then iterate and move the data
>>>> (deprecate
>>>> the older feature if necessary, e.g. If that decision comes after
>>>> Icehouse, else of a different/better implementation comes before
>>>> Icehouse,
>>>> then no harm done).
>>> -1
>>>
>>>>>> There is precedence for an optional metadata section that can
>>>>>>contain
>>>>>> any
>>> As Clint said, you're comparing APIs to templates. Those are very
>>> different.
>>>
>>>>>> end-user data in other Openstack projects and it is necessary in
>>>>>>order
>>>>>> to
>>>>>> iterate quickly and provide value to Heat.
>>> Non sequitur.
>>>
>>>>> Nobody has said you can't have meta-data on stacks, which is what
>>>>>other
>>>>> projects use.
>>>>>
>>>>>> There are many use cases that can be discussed here, but I wanted to
>>>>>> reiterate an initial discussion point that, by definition,
>>>>>> "stack/template_metadata" does not have any hard requirements in
>>>>>>terms
>>>>>> of
>>>>>> schema or what does or does not belong in it.
>>>>>>
>>>>>> One of the initial use cases is to allow template authors to
>>>>>> categorize
>>>>>> the template as a specific "type".
>>>>>>
>>>>>>      template_metadata:
>>>>>>          short_description: Wordpress
>>> First of all, "short_description" is a dreadful name for something that
>>> is basically tags.
>>>
>>> Secondly, these are tags in the template catalog. So why are they
>>> incorporated in the template, and not stored in the template catalog?
>>>
>>> Finally, if you really want this it can be trivially implemented using
>>> comments.
>>>
>>>>> Interesting. Would you support adding a "category" keyword to python
>>>>>so
>>>>> we don't have to put it in setup.cfg and so that the egg format
>>>>>doesn't
>>>>> need that section? Pypi can just parse the python to categorize the
>>>>> apps
>>>>> when they're uploaded. We could also have a file on disk for qcow2
>>>>> images
>>>>> that we upload to glance that will define the meta-data.
>>>>>
>>>>> To be more direct, I don't think the templates themselves are where
>>>>> this
>>>>> meta-data belongs. A template is self-aware by definition, it doesn't
>>>>> need the global metadata section to tell it that it is WordPress. For
>>>>> anything else that needs to be globally referenced there are
>>>>> parameters.
>>>>> Having less defined inside the template means that you get _more_
>>>>> wiggle
>>>>> room for your template repository.
>>>> Clint, you are correct that the Template does not need to know what it
>>>> is.
>>>>   It's every other service (and users of those services) that a
>>>>Template
>>>> passes through or to that would care to know what it is. We are
>>>> suggesting
>>>> we put that meta data in the template file and expressly ignore it for
>>>> purposes of parsing the template language in the Heat engine, so we
>>>> agree
>>>> it not a necessary part of the template.  Sure, we could encode the
>>>> metadata info in a separate catalog...  but, take the template out of
>>>> the
>>>> catalog and now all that useful associated data is lost or would need
>>>>to
>>>> be recreated by someone or some service.  That does not make the
>>>> template
>>>> portable, and that is a key aspect of what we are trying to achieve
>>>>(all
>>>> user-facing clients, like Horizon, or humans reading the file, can
>>>>take
>>>> advantage). We don't entirely know yet what is most useful in
>>>> portability
>>> What's most useful in portability is placing the minimum requirements
>>> (especially undocumented and/or non-standard requirements) on the
>>> template author, since you can't control the template author.
>>>
>>>> and what isn't, so meta data in-template provides the "wiggle room"
>>>> innovation space to suss that out.  We already know of some specific
>>>>use
>>>> cases of data that we feel are important, which Tim identified one
>>>> specific example.. As specific metadata items become popular or prove
>>>>to
>>>> be useful to rely on by the larger community or service operators
>>>> (public
>>>> and private) of Heat, we as a community can drive that information
>>>>back
>>>> into the schema for the template or some portable format mechanism.
>>>>
>>>>> I 100% support having a template catalog. IMO it should be glance,
>>>>> which is our catalog service in OpenStack. Who cares if nova or heat
>>>>> are
>>>>> consuming images or templates. It is just sharable blobs of data and
>>>>> meta-data in a highly scalable service. It already has the concept of
>>>>> global and tenant-scope. It just needs an image type of 'hot' and
>>>>>then
>>>>> heat can start consuming templates from glance. And the template
>>>>> authors
>>>>> should maintain some packaging meta-data in glance to communicate to
>>>>> users that this is "Wordpress" and "Single-Node". If Glance's
>>>>>meta-data
>>>>> is too limiting, expand it! I'm sure image authors and consumers
>>>>>would
>>>>> appreciate that.
>>>> This is definitely interesting... And takes the long view IMO.  Let me
>>>> explain:  I don't anticipate Heat catalog'ing in Glance is something
>>>> that
>>>> has a high chance of getting implemented in the Icehouse timeframe (at
>>>> least, not more so than in-template metadata), do you?  From a SOA
>>>> service
>>>> deployer perspective, I'm sure you can appreciate that rolling out new
>>>> functionality in Glance to support an Orchestration project use case
>>>>is
>>>> not simple, and requires strong business justification and
>>>>coordination
>>>> as
>>>> an operator of a cloud.. One worth exploring for sure, but not the
>>>>go-to
>>>> default strategy. I view the metadata change as very minor with little
>>>> to
>>>> no disruption to any service, including Heat (Heat just ignores the
>>> I view it as an open invitation to fragment the template format and the
>>> ecosystem.
>>>
>>>> metadata, completely).. This fits very well in an iterative
>>>>development
>>>> model.  New blueprints could be raised, as you suggested, to move
>>>> metadata
>>>> and catalog features into Glance.  My concern is that if we go the
>>>> Glance
>>>> route now, we are encouraging a precedent that we aren't iterative
>>>>(due
>>>> to
>>>> abandoning an already accepted blueprint in favor of a more complex
>>>>and
>>>> time intensive solution) and that we won't get this implemented within
>>>> the
>>>> current release cycle.
>>>>
>>>>>> This would let the client of the Heat API group the templates by
>>>>>>type
>>>>>> which would create a better user experience when selecting or
>>>>>>managing
>>>>>> templates. The the end-user could select "Wordpress" and drill down
>>>>>> further to select templates with different options, "single node",
>>>>>>"2
>>>>>> web
>>>>>> nodes", etc...
>>>>>>
>>>>> That is all api stuff, not language stuff.
>>>> If this were done solely at the API, it would have to be maintained
>>>> 1-to-1
>>>> with a template (in which case there is an implicit and explicit
>>>> association), and exported with the template in order to port the
>>>> template.
>>> I don't understand all this stuff about 'portability'. From your
>>> previous descriptions of the template catalog, it sounded like it's for
>>> templates hand-picked by the operator. Why would one operator want to
>>> import the tags defined by another operator anyway? And Why wouldn't
>>> that just be a git clone of the backing repository? And how does any of
>>> this make the template more "portable"???
>>>
>>>>>> Once a feature has consistently proven that it adds value to Heat or
>>>>>> Horizon, then I would suggest that we can discuss the schema for
>>>>>>that
>>>>>> feature and codify it then.
>>>>>>
>>>>>> In order to keep the discussion simple, I am only responding to the
>>>>>> need
>>>>>> for stack/template metadata at the moment but I'm sure discussions
>>>>>>on
>>>>>> the
>>>>>> management api and template catalog will follow.
>>>>>>
>>>>> Your example puts the template catalog in front of this feature, and
>>>>>I
>>>>> think that exposes this feature as misguided.
>>>> I'd like template author and creation date portable so that it goes
>>>> wherever the template goes in a portable format that consumers (any
>>>> client) of the Heat service can understand.  I'd also like this
>>> Put it in a comment.
>>>
>>>> information to be available for a deployed stack in Heat, so it is
>>>>need
>>>> not be catalog specific.  If author/creation-date information were
>>>>only
>>> If it needs to be available for a deployed stack (you haven't made an
>>> argument for this yet) then add it into the template-proper. If Heat
>>> needs to read it then some sort of wild-west schemaless metadata area
>>>is
>>> definitely not the place to put it.
>>>
>>> Alternately, maybe we need to look at storing templates in such a way
>>> that the comments are preserved. I would definitely look favourably on
>>>a
>>> blueprint to that effect.
>>>
>>>> available in the catalog, then we would have to entirely wrap the Heat
>>>> API
>>>> in another service or risk drift in the data between Heat and the
>>>> separate
>>>> non-wrapping catalog service. Instead of raising a blueprint for every
>>>> piece of data that may be useful to deploying this service (some of
>>>> which
>>> Please don't raise a blueprint for every patch. e.g. a feature and its
>>> documentation 
>>> 
>>>(https://blueprints.launchpad.net/heat/+spec/update-stack-metadata-spec)
>>> are not two blueprints.
>>>
>>>> certain folks may not care about [e.g. Many of the folks arguing
>>>>against
>>>> metadata all together], the in-template metadata is, IMO, a suitable
>>>> approach to experiment and then we can drive back experiential
>>>>use-cases
>>>> into future improvements (e.g. in-language schema changes or catalog
>>>> changes).  At Rackspace we have experience running a nearly identical
>>>> service, so we know this data, in this particular place, is a valid
>>>>use
>>>> case for consumers of an Orchestration service.
>>>> Example:
>>>> (1) Cloud customer of service operator deploys a template (may or may
>>>> not
>>>> have come from the catalog)
>>> If it didn't come from the catalog provided by the operator, what are
>>> the chances that it contains this information anyway? How is the
>>>average
>>> template author going to know about the schema for the metadata read by
>>> your proprietary tool?
>>>
>>>> (2) Customer encounters problem with the stack and calls support
>>>> (3) Support specialist [not the service operator/developer] is trying
>>>>to
>>>> figure out who to contact for additional help given failures seen with
>>>> the
>>>> stack. The dev team of the service is not the expert in application
>>>>foo
>>>> and did not develop the template. It's not a Heat issue, it's a
>>>> template/application foo issue.
>>> This sounds like a good argument for Heat preserving comments.
>>>
>>>> Having this detail come from Glance, as you pointed out, would be
>>>>great.
>>>> But, my hope is that many folks will develop and share templates, and
>>>> the
>>>> folks developing templates won't necessarily be running glance or have
>>>> access to upload their own images into glance (that is a service
>>>> operator
>>>> choice).  If we force this basic info to glance, we are limiting our
>>>> portability (and therefore adoption story) of Heat.
>>> What's really going to limit portability is having multiple overlapping
>>> undocumented proprietary schemas for parts of the template. I actually
>>> think this has the potential to damage the Heat ecosystem, and that's
>>> one of the reasons I'm pushing back on it.
>>>
>>> regards,
>>> Zane.
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev