Open Stack

Timur,

I am excited to hear that you think that Barricade could be used in
Merlin! Your feedback is fantastic and I am impressed that you have been
able to understand Barricade so well from only the source and the spec.


I am replying to your feedback inline:

On 8/29/14, 2:59 PM, "Timur Sufiev" <tsufiev at mirantis.com> wrote:

>Drago,
>
>It sounds like you convinced me to give D3.js a second chance :). I'll
>experiment with what can be achieved using force-directed graph layout
>combined with some composable svg object, hopefully this will save me
>from placing objects on the canvas on my own.
>
>I've read the barricade_Spec.js several times and part of
>barricade.js. Code is very interesting and allowed me to refresh some
>JavaScript knowledge I used a looong ago :). The topic I definitely
>haven't fully grasped is deferred/deferrable/referenced objects. The
>thing I've understood is that if some scheme includes '@ref' key, then
>it tries to get value returned from the resolver function no matter
>what value has provided during scheme instantiation. Am I right? Is
>'needs' section required for the value to be resolved? The examples
>in file with tests are a bit mind-bending, so I failed to imagine how
>it works for real use-cases. Also I'm interested whether it is
>possible to define a schema that allows both to provide the value
>directly and via reference?

Reference resolving is probably the most complicated part of Barricade
right now, and I think that it should be more simple and clear than it is.

You are correct about resolver, except that the intent is to use the value
provided to find the correct reference. The spec is not a good real-world
example. To illustrate how @ref should really be used, I will use
intrinsic functions in a Heat template as an example. Take for instance
get_resource:

{get_resource: “some_resource_id”}

Here, “some_resource_id” is a just a string, but it is referring to an
actual resource defined elsewhere (the resource section of the template).
To resolve this string into the actual resource, @ref can be used in this
way:

'get_resource': {
    '@type': String,
    '@ref': {
        to: function () { return hot.Resource; },
        needs: function () { return hot.Template; },
        resolver: function (json, template) {
            return template.get('resources').getByID(json.get());
        }
    }
        }

This can be read as “get_resource is a string which refers to a Resource,
and it needs a Template in order to find what it’s referencing”. The idea
is that both the value that is supposed to be a reference and the actual
reference will have the same parent somewhere up the chain. The “needs”
property describes how far up the chain to go to find that parent and the
resolver is then used to find the needed reference somewhere within that
parent. Here, the resolver is called once the Template is encountered, and
the reference is found in the resources section of the template by using
the string that was originally provided.


> Among other things that inclined me to
>give some feedback are:
>* '@type' vs. '@class' - is the only difference between them that
>'@type' refers to primitive type and '@class' refers to Barricade.js
>scheme? Perhaps it could be reduced to a single word to make things
>simpler?

The reason for separate tags was that I was not sure if they were truly
mutually exclusive. Right now, @class causes @type (among other things) to
be ignored, because that information should be contained in the class that
@class is referring to. Therefore, I think that the two tags could
definitely be combined into one.

>* '?' vs '*' - seems they are used in different contexts, '?' is for
>Object and '*' for Array - are 2 distinct markers actually needed?

No, you’re right. It was simply to help separate mutable objects from
arrays. It is not necessary.

>* Is it better for objects with fixed schema to fail when unexpected
>key is passed to them? Currently they don't.

For set(), it does fail and give an error (though only in the console
right now) [1], but for get(), it does not give any warning or error [2],
so that could be changed.

>* Pushing an element of wrong type into an arraylike scheme still
>creates an element with empty default value.

Yes, this should be fixed relatively soon now that better validation has
been added. I think that it should simply fail instead.

>* Is it possible to create schemas with arbitrary default values (the
>example from spec caused me to think that default values cannot be
>specified)?

It should be possible, but it is not yet.

>* 'required' property does not really force such key to be provided
>during schema instantiation - I presume this is going to change when
>the real validation arrives?

Yes.

>* What are the conceptual changes between objects instantiated from
>mutable (with '?') and from immutable (with fixed keys) schema?

“Mutable” objects (for lack of a better name) are represented internally
as arrays with each element having an ID. Since each key of a mutable
object is user-defined, it makes more sense to have the key as a member of
the value it belongs to. This way, the key can be changed via the value
itself so that code operating on the values does not need to have a
reference to its container as well. This also solves the problem of having
values that exist in multiple containers/places, such as the @ref example
above. Immutable objects on the other hand work as you’d expect them to,
where the values of the keys can be set but the keys themselves cannot be
changed and therefore are not a part of the value in any way.

>Thank you very much for your efforts, I think that Barricade.js could
>provide a solid foundation for Merlin!

Thank you,
Drago

[1] 
https://github.com/rackerlabs/barricade/blob/master/src/immutable_object.js
#L31
[2] 
https://github.com/rackerlabs/barricade/blob/master/src/immutable_object.js
#L26

>
>On Thu, Aug 28, 2014 at 9:31 PM, Drago Rosson
><drago.rosson at rackspace.com> wrote:
>> Timur,
>>
>> Composable entities can be a real need for Heat if provider templates
>> (which allow templates to be used as a resource, with a template’s
>> parameters and outputs becoming properties and attributes, respectively)
>> are to be included in the app. A provider template resource, since it
>>is a
>> template itself, would be composed of resources which would require a
>> composable entity. What is great about D3’s force graph is that it’s
>>nodes
>> and links can be completely arbitrary - meaning they can be any
>>JavaScript
>> object (including an SVG or DOM element). Additionally, the force graph
>> simulation updates x and y properties on those elements and calls a
>> user-defined “tick” function. The tick function can use the x and y
>> properties in any way it wants to do the *actual* update to the position
>> of each element. For example, this is how multiple foci can be
>>implemented
>> [1]. Lots of other customization is available, including starting and
>> stopping the simulation, updating the node and link data, and having
>> per-element control of most (all?) properties such as charge or link
>> distance.
>>
>> Composability can be achieved using SVG’s <g> elements to group multiple
>> graphical elements together. The tick function would need to update the
>> <g>’s transform attribute [2]. This is how it is done in my app since my
>> nodes and links are composed of icons, labels, backgrounds, etc. I think
>> that D3’s force graph is not a limiting factor since it itself does not
>> concern itself with graphics at all. Therefore, the question seems to be
>> whether D3 can do everything graphically that Merlin needs. D3 is not a
>> graphics API, but it does have support for graphical manipulation,
>> animations, and events. They have sufficed for me so far. Plus, D3 can
>>do
>> these things without having to use its fancy data transformations so it
>> can be used as a low-level SVG library where necessary. D3 can do a lot
>> [3] so hopefully it could also do what Merlin needs.
>>
>> You are in luck, because I have just now open-sourced Barricade! Check
>>it
>> out [4]. I am working on getting documentation written for it but to see
>> some ways it can be used, look at its test suite [5].
>>
>> [1] http://bl.ocks.org/mbostock/1021953
>> [2] node.attr("transform", function (d) {
>>         return "translate(" + d.x + ", " + d.y + ")";
>>     });
>> [3] http://christopheviau.com/d3list/
>> [4] https://github.com/rackerlabs/barricade
>>
>> [5]
>> 
>>https://github.com/rackerlabs/barricade/blob/master/test/barricade_Spec.j
>>s
>>
>> On 8/28/14, 10:03 AM, "Timur Sufiev" <tsufiev at mirantis.com> wrote:
>>
>>>Hello, Drago!
>>>
>>>I'm extremely interested in learning more about your HOT graphical
>>>builder. The screenshots you had attached look gorgeous! Yours visual
>>>representation of Heat resources is much more concise and simple than
>>>I had drawn in Merlin PoC mock-ups [1]. On the other hand I have some
>>>suspicions that D3.js is a good fit for a general purpose UI toolkit
>>>Merlin aims to provide. Please don't get me wrong, D3.js is a great
>>>library which can do fantastic things with data - in case your
>>>data<->visualization use-case maps to the one of the facilities D3.js
>>>provides out of the box. In case it doesn't, there are 2 options:
>>>either change your approach to what should be visualized/how it should
>>>be visualized, or tweak some inner machinery of D3.js
>>>
>>>While bending the design towards the facilities of D3.js doesn't seem
>>>a viable choice, changing D3.js from inside can be painful too. AFAIK
>>>force-directed graph layout from D3.js doesn't provide the means to
>>>represent composable entities (which isn't a big problem for Heat, but
>>>is a very serious issue for Murano) out of the box. By composable I
>>>mean something like [2] - but with much more complex inner structure
>>>(imagine the Resource entity [3] having as its properties other
>>>Resource entities which are shown as simple rounded rectangles with
>>>labels on that picture, but are expanded into complex objects similar
>>>to [3] once the user, say, clicks on them). As far as I understand,
>>>you are visualizing that kind of composition via arrow links, but I'd
>>>like to try another design options (especially in case of Murano) and
>>>fear that D3.js will constrain me here. I've been thinking a bit about
>>>using more low-level SVG js-framework like Raphael.js - it doesn't
>>>offer most of the goodies D3.js does, but also it doesn't force me to
>>>create the design based on some data transformations in a way that
>>>D3.js does, providing the good old procedural API instead. Of course,
>>>I may be wrong, perhaps more time and efforts invested into Merlin PoC
>>>would allow me to realize it (or not).
>>
>>>Yet you are totally right having stressed the importance of right tool
>>>for implementing the underlying object model (or JSON-wrapper as you
>>>called it) - Barricade.js. That's the second big part of work Merlin
>>>had to do, and I couldn't underestimate how it would be beneficial for
>>>Merlin to leverage some of the facilities that Barricade.js provides.
>>>I'll gladly look at the demo of template builder and Barricade. Is
>>>there any chance I could take a look also at the source code of
>>>Barricade.js, so I would better understand to which extent it suits
>>>Merlin’s needs? I've searched through github.com and didn't found any
>>>traces of Barricade.js repo, so it seems like some in-house project to
>>>me. What are your plans for sharing this library with the community?
>>>
>>>[1] https://wiki.openstack.org/wiki/Merlin/PoC
>>>[2] http://mbostock.github.io/d3/talk/20111116/pack-hierarchy.html
>>>[3] https://wiki.openstack.org/wiki/File:Merlin_poc_3.png
>>>
>>>On Wed, Aug 27, 2014 at 6:14 PM, Drago Rosson
>>><drago.rosson at rackspace.com> wrote:
>>>> Hello Timur,
>>>>
>>>> I have been developing a graphical Heat Orchestration Template
>>>>builder.
>>>>I
>>>> first heard about the Merlin project around a week ago and found that
>>>>what
>>>> I have developed in the past few months is in line with what Merlin is
>>>> trying to accomplish.
>>>>
>>>> Some of the features of the template builder (described further down)
>>>> could be used as part of the UI framework Merlin aims to produce.
>>>>However,
>>>> its most important and useful component is a JavaScript library I have
>>>> developed called Barricade.js, which powers the template builder’s
>>>>data
>>>> layer.
>>>>
>>>> Barricade.js aims to solve the problem of using JSON data across a web
>>>> app. It creates data model objects out of JSON using a predefined
>>>>schema
>>>> (which looks similar to the schema used in your Murano Dynamic UI).
>>>>This
>>>> removes the disadvantages with JSON and introduces some very important
>>>> advantages:
>>>>
>>>> - Encapsulation. JSON values can be set to any type or deleted
>>>>entirely,
>>>> which either causes errors when UI components expect these values to
>>>>exist
>>>> or be of a certain type, or forces the UI components to constantly
>>>>check
>>>> for correctness. Barricade instead wraps around the JSON and provides
>>>> accessor methods to ensure type-safe data manipulation. Additionally,
>>>> Barricade objects are observable, so changes made to their data
>>>>trigger
>>>> events that can be subscribed to by UI components.
>>>>
>>>> - Normalization. Whenever properties that are defined in the schema
>>>>are
>>>> missing in the JSON, Barricade will fill them in with default values.
>>>>This
>>>> way, UIs will always have valid values where they expect them, making
>>>> their design much simpler. Optional properties are extremely common in
>>>> Heat templates.
>>>>
>>>> - Metadata. Anything extra attached to JSON must be handled carefully
>>>> (such as when converting back to the original YAML format). By
>>>>wrapping
>>>> the JSON with Barricade, metadata and convenience methods that UI
>>>> components can use can be defined. For instance, the datatype of any
>>>>value
>>>> or a description to go along with each property in a Heat resource.
>>>>
>>>> - Validation. Soon, the schema will allow defining validators that
>>>>will
>>>> run whenever a new value is attempted to be set on data. Messages
>>>>about
>>>> failed validation will be available so that the UI can display it.
>>>>This
>>>> system seems to be very similar to dynamic UI’s.
>>>>
>>>> What this all boils down to is that all of the logic required to
>>>>ensure
>>>> JSON’s integrity is rolled into Barricade instead of sprinkled into
>>>>all
>>>>of
>>>> the UI code. This way, UI components can be confident in the data that
>>>> they are working with, which makes their code more concise and faster
>>>>to
>>>> develop.
>>>>
>>>> Barricade seems like a great fit for Merlin because the projects it
>>>> targets (Heat, Solum, Mistral, Murano) use YAML files that can be used
>>>> with Barricade once they are converted to JSON.
>>>>
>>>>
>>>> About the template builder (see screenshots attached):
>>>>
>>>> - Uses an interactive canvas (powered by a D3.js force-directed graph)
>>>>to
>>>> display Heat resources and the dependencies in between them. New
>>>>resources
>>>> can be drag-and-dropped from a panel onto the canvas. Different
>>>>resource
>>>> types are attracted to different respective areas so that they
>>>> automatically arrange themselves in a familiar network topology
>>>>hierarchy.
>>>>  A form/panel is displayed when a resource on the canvas is clicked so
>>>> that the resource’s properties can be edited.
>>>>
>>>> - Has two―way data binding provided by Knockout.js interfacing with
>>>> Barricade.js so that editing values in forms automatically updates the
>>>> topology on the canvas (e.g. Changing resource IDs or
>>>>creating/removing
>>>> dependencies between resources).
>>>>
>>>> - Allows for direct-editing of the template (via text input) and
>>>>loading
>>>> templates via URLs.
>>>>
>>>>
>>>> Please, let me know if you would like a demo of the template builder
>>>>and
>>>> Barricade!
>>>>
>>>> Thanks,
>>>> Drago Rosson
>>>>
>>>>
>>>>
>>>>
>>>> On 8/18/14, 5:19 AM, "Timur Sufiev" <tsufiev at mirantis.com> wrote:
>>>>
>>>>>David,
>>>>>
>>>>>I'm happy to hear that :)! After thinking a bit, I came up with the
>>>>>following strategy for further Merlin development: make all the
>>>>>commits into a separate repository (stackforge/merlin) at least until
>>>>>the PoC is ready. This will allow to keep project history more
>>>>>granular instead of updating one large commit inside openstack/horizon
>>>>>gerrit (thus also lessening the burden on Horizon reviewers). Once the
>>>>>Merlin proceeds from the experimental/PoC phase to the implementing of
>>>>>a more elaborated spec, it will be just the time for it to join with
>>>>>the Horizon.
>>>>>
>>>>>On Wed, Aug 13, 2014 at 2:48 AM, Lyle, David <david.lyle at hp.com>
>>>>>wrote:
>>>>>> On 8/6/14, 1:41 PM, "Timur Sufiev" <tsufiev at mirantis.com> wrote:
>>>>>>
>>>>>>>Hi, folks!
>>>>>>>
>>>>>>>Two months ago there was an announcement in ML about gathering the
>>>>>>>requirements for cross-project UI library for
>>>>>>>Heat/Mistral/Murano/Solum [1]. The positive feedback in related
>>>>>>>googledoc [2] and some IRC chats and emails that followed convinced
>>>>>>>me
>>>>>>>that I'm not the only person interested in it :), so I'm happy to
>>>>>>>make
>>>>>>>the next announcement.
>>>>>>>
>>>>>>>The project finally has got its name - 'Merlin' (making complex UIs
>>>>>>>is
>>>>>>>a kind of magic), Openstack wiki page [3] and all other stuff like
>>>>>>>stackforge repo, launchpad page and IRC channel (they are all
>>>>>>>referenced in [3]). For those who don't like clicking the links,
>>>>>>>here
>>>>>>>is quick summary.
>>>>>>>
>>>>>>>Merlin aims to provide a convenient client side framework for
>>>>>>>building
>>>>>>>rich UIs for Openstack projects dealing with complex input data with
>>>>>>>lot of dependencies and constraints (usually encoded in YAML format
>>>>>>>via some DSL) - projects like Heat, Murano, Mistral or Solum. The
>>>>>>>ultimate goal for such UI is to save users from reading
>>>>>>>comprehensive
>>>>>>>documentation just in order to provide correct input data, thus
>>>>>>>making
>>>>>>>the UI of these projects more user-friendly. If things go well for
>>>>>>>Merlin, it could be eventually merged into Horizon library (I¹ll
>>>>>>>spare
>>>>>>>another option for the end of this letter).
>>>>>>>
>>>>>>>The framework trying to solve this ambitious task is facing at
>>>>>>>least 2
>>>>>>>challenges:
>>>>>>>(1) enabling the proper UX patterns and
>>>>>>>(2) dealing with complexities of different projects' DSLs.
>>>>>>>
>>>>>>>Having worked on DSL things in Murano project before, I'm planning
>>>>>>>at
>>>>>>>first to deal with the challenge (2) in the upcoming Merlin PoC. So,
>>>>>>>here is the initial plan: design an in-framework object model (OM)
>>>>>>>that could translated forth and back into target project's DSL. This
>>>>>>>OM is meant to be synchronised with visual elements shown on browser
>>>>>>>canvas. Target project is the Heat with its HOT templates - it has
>>>>>>>the
>>>>>>>most well-established syntax among other projects and comprehensive
>>>>>>>documentation.
>>>>>>>
>>>>>>>Considering the challenge (1), not being a dedicated UX engineer,
>>>>>>>I'm
>>>>>>>planning to start with some rough UI concepts [4] and gradually
>>>>>>>improve them relying on community feedback, and especially,
>>>>>>>Openstack
>>>>>>>UX group. If anybody from the UX team (or any other team!) is
>>>>>>>willing
>>>>>>>to be involved to a greater degree than just giving some feedback,
>>>>>>>you're are enormously welcome! Join Merlin, it will be fun :)!
>>>>>>>
>>>>>>>Finally, with this announcement I¹d like to start a discussion with
>>>>>>>Horizon community. As far as I know, Horizon in its current state
>>>>>>>lacks such UI toolkit as Merlin aims to provide. Would it be by any
>>>>>>>chance possible for the Merlin project to be developed from the very
>>>>>>>beginning as part of Horizon library? This choice has its pros and
>>>>>>>cons I¹m aware of, but I¹d like to hear the opinions of Horizon
>>>>>>>developers on that matter.
>>>>>>
>>>>>> I would like to see this toolset built into Horizon. That will make
>>>>>>it
>>>>>> accessible to integrated projects like Heat that Horizon already
>>>>>>supports,
>>>>>> but will also allow other projects to use the horizon library as a
>>>>>> building block to providing managing project specific DSLs.
>>>>>>
>>>>>> David
>>>>>>
>>>>>>>
>>>>>>>[1]
>>>>>>>http://lists.openstack.org/pipermail/openstack-dev/2014-June/037054.
>>>>>>>ht
>>>>>>>ml
>>>>>>>[2]
>>>>>>>https://docs.google.com/a/mirantis.com/document/d/19Q9JwoO77724RyOp7
>>>>>>>Xk
>>>>>>>pY
>>>>>>>mA
>>>>>>>Lwmdb7JjoQHcDv4ffZ-I/edit#
>>>>>>>[3] https://wiki.openstack.org/wiki/Merlin
>>>>>>>[4] https://wiki.openstack.org/wiki/Merlin/SampleUI
>>>>>>>
>>>>>>>--
>>>>>>>Timur Sufiev
>>>>>>>
>>>>>>>_______________________________________________
>>>>>>>OpenStack-dev mailing list
>>>>>>>OpenStack-dev at lists.openstack.org
>>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>
>>>>>--
>>>>>Timur Sufiev
>>>>>
>>>>>_______________________________________________
>>>>>OpenStack-dev mailing list
>>>>>OpenStack-dev at lists.openstack.org
>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>>
>>>--
>>>Timur Sufiev
>>>
>>>_______________________________________________
>>>OpenStack-dev mailing list
>>>OpenStack-dev at lists.openstack.org
>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>-- 
>Timur Sufiev
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev