Open Stack

Drago,

It sounds like you convinced me to give D3.js a second chance :). I'll
experiment with what can be achieved using force-directed graph layout
combined with some composable svg object, hopefully this will save me
from placing objects on the canvas on my own.

I've read the barricade_Spec.js several times and part of
barricade.js. Code is very interesting and allowed me to refresh some
JavaScript knowledge I used a looong ago :). The topic I definitely
haven't fully grasped is deferred/deferrable/referenced objects. The
thing I've understood is that if some scheme includes '@ref' key, then
it tries to get value returned from the resolver function no matter
what value has provided during scheme instantiation. Am I right? Is
'needs' section required for the value to be resolved? The examples
in file with tests are a bit mind-bending, so I failed to imagine how
it works for real use-cases. Also I'm interested whether it is
possible to define a schema that allows both to provide the value
directly and via reference? Among other things that inclined me to
give some feedback are:
* '@type' vs. '@class' - is the only difference between them that
'@type' refers to primitive type and '@class' refers to Barricade.js
scheme? Perhaps it could be reduced to a single word to make things
simpler?
* '?' vs '*' - seems they are used in different contexts, '?' is for
Object and '*' for Array - are 2 distinct markers actually needed?
* Is it better for objects with fixed schema to fail when unexpected
key is passed to them? Currently they don't.
* Pushing an element of wrong type into an arraylike scheme still
creates an element with empty default value.
* Is it possible to create schemas with arbitrary default values (the
example from spec caused me to think that default values cannot be
specified)?
* 'required' property does not really force such key to be provided
during schema instantiation - I presume this is going to change when
the real validation arrives?
* What are the conceptual changes between objects instantiated from
mutable (with '?') and from immutable (with fixed keys) schema?

Thank you very much for your efforts, I think that Barricade.js could
provide a solid foundation for Merlin!

On Thu, Aug 28, 2014 at 9:31 PM, Drago Rosson
<drago.rosson at rackspace.com> wrote:
> Timur,
>
> Composable entities can be a real need for Heat if provider templates
> (which allow templates to be used as a resource, with a template’s
> parameters and outputs becoming properties and attributes, respectively)
> are to be included in the app. A provider template resource, since it is a
> template itself, would be composed of resources which would require a
> composable entity. What is great about D3’s force graph is that it’s nodes
> and links can be completely arbitrary - meaning they can be any JavaScript
> object (including an SVG or DOM element). Additionally, the force graph
> simulation updates x and y properties on those elements and calls a
> user-defined “tick” function. The tick function can use the x and y
> properties in any way it wants to do the *actual* update to the position
> of each element. For example, this is how multiple foci can be implemented
> [1]. Lots of other customization is available, including starting and
> stopping the simulation, updating the node and link data, and having
> per-element control of most (all?) properties such as charge or link
> distance.
>
> Composability can be achieved using SVG’s <g> elements to group multiple
> graphical elements together. The tick function would need to update the
> <g>’s transform attribute [2]. This is how it is done in my app since my
> nodes and links are composed of icons, labels, backgrounds, etc. I think
> that D3’s force graph is not a limiting factor since it itself does not
> concern itself with graphics at all. Therefore, the question seems to be
> whether D3 can do everything graphically that Merlin needs. D3 is not a
> graphics API, but it does have support for graphical manipulation,
> animations, and events. They have sufficed for me so far. Plus, D3 can do
> these things without having to use its fancy data transformations so it
> can be used as a low-level SVG library where necessary. D3 can do a lot
> [3] so hopefully it could also do what Merlin needs.
>
> You are in luck, because I have just now open-sourced Barricade! Check it
> out [4]. I am working on getting documentation written for it but to see
> some ways it can be used, look at its test suite [5].
>
> [1] http://bl.ocks.org/mbostock/1021953
> [2] node.attr("transform", function (d) {
>         return "translate(" + d.x + ", " + d.y + ")";
>     });
> [3] http://christopheviau.com/d3list/
> [4] https://github.com/rackerlabs/barricade
>
> [5]
> https://github.com/rackerlabs/barricade/blob/master/test/barricade_Spec.js
>
> On 8/28/14, 10:03 AM, "Timur Sufiev" <tsufiev at mirantis.com> wrote:
>
>>Hello, Drago!
>>
>>I'm extremely interested in learning more about your HOT graphical
>>builder. The screenshots you had attached look gorgeous! Yours visual
>>representation of Heat resources is much more concise and simple than
>>I had drawn in Merlin PoC mock-ups [1]. On the other hand I have some
>>suspicions that D3.js is a good fit for a general purpose UI toolkit
>>Merlin aims to provide. Please don't get me wrong, D3.js is a great
>>library which can do fantastic things with data - in case your
>>data<->visualization use-case maps to the one of the facilities D3.js
>>provides out of the box. In case it doesn't, there are 2 options:
>>either change your approach to what should be visualized/how it should
>>be visualized, or tweak some inner machinery of D3.js
>>
>>While bending the design towards the facilities of D3.js doesn't seem
>>a viable choice, changing D3.js from inside can be painful too. AFAIK
>>force-directed graph layout from D3.js doesn't provide the means to
>>represent composable entities (which isn't a big problem for Heat, but
>>is a very serious issue for Murano) out of the box. By composable I
>>mean something like [2] - but with much more complex inner structure
>>(imagine the Resource entity [3] having as its properties other
>>Resource entities which are shown as simple rounded rectangles with
>>labels on that picture, but are expanded into complex objects similar
>>to [3] once the user, say, clicks on them). As far as I understand,
>>you are visualizing that kind of composition via arrow links, but I'd
>>like to try another design options (especially in case of Murano) and
>>fear that D3.js will constrain me here. I've been thinking a bit about
>>using more low-level SVG js-framework like Raphael.js - it doesn't
>>offer most of the goodies D3.js does, but also it doesn't force me to
>>create the design based on some data transformations in a way that
>>D3.js does, providing the good old procedural API instead. Of course,
>>I may be wrong, perhaps more time and efforts invested into Merlin PoC
>>would allow me to realize it (or not).
>
>>Yet you are totally right having stressed the importance of right tool
>>for implementing the underlying object model (or JSON-wrapper as you
>>called it) - Barricade.js. That's the second big part of work Merlin
>>had to do, and I couldn't underestimate how it would be beneficial for
>>Merlin to leverage some of the facilities that Barricade.js provides.
>>I'll gladly look at the demo of template builder and Barricade. Is
>>there any chance I could take a look also at the source code of
>>Barricade.js, so I would better understand to which extent it suits
>>Merlin’s needs? I've searched through github.com and didn't found any
>>traces of Barricade.js repo, so it seems like some in-house project to
>>me. What are your plans for sharing this library with the community?
>>
>>[1] https://wiki.openstack.org/wiki/Merlin/PoC
>>[2] http://mbostock.github.io/d3/talk/20111116/pack-hierarchy.html
>>[3] https://wiki.openstack.org/wiki/File:Merlin_poc_3.png
>>
>>On Wed, Aug 27, 2014 at 6:14 PM, Drago Rosson
>><drago.rosson at rackspace.com> wrote:
>>> Hello Timur,
>>>
>>> I have been developing a graphical Heat Orchestration Template builder.
>>>I
>>> first heard about the Merlin project around a week ago and found that
>>>what
>>> I have developed in the past few months is in line with what Merlin is
>>> trying to accomplish.
>>>
>>> Some of the features of the template builder (described further down)
>>> could be used as part of the UI framework Merlin aims to produce.
>>>However,
>>> its most important and useful component is a JavaScript library I have
>>> developed called Barricade.js, which powers the template builder’s data
>>> layer.
>>>
>>> Barricade.js aims to solve the problem of using JSON data across a web
>>> app. It creates data model objects out of JSON using a predefined schema
>>> (which looks similar to the schema used in your Murano Dynamic UI). This
>>> removes the disadvantages with JSON and introduces some very important
>>> advantages:
>>>
>>> - Encapsulation. JSON values can be set to any type or deleted entirely,
>>> which either causes errors when UI components expect these values to
>>>exist
>>> or be of a certain type, or forces the UI components to constantly check
>>> for correctness. Barricade instead wraps around the JSON and provides
>>> accessor methods to ensure type-safe data manipulation. Additionally,
>>> Barricade objects are observable, so changes made to their data trigger
>>> events that can be subscribed to by UI components.
>>>
>>> - Normalization. Whenever properties that are defined in the schema are
>>> missing in the JSON, Barricade will fill them in with default values.
>>>This
>>> way, UIs will always have valid values where they expect them, making
>>> their design much simpler. Optional properties are extremely common in
>>> Heat templates.
>>>
>>> - Metadata. Anything extra attached to JSON must be handled carefully
>>> (such as when converting back to the original YAML format). By wrapping
>>> the JSON with Barricade, metadata and convenience methods that UI
>>> components can use can be defined. For instance, the datatype of any
>>>value
>>> or a description to go along with each property in a Heat resource.
>>>
>>> - Validation. Soon, the schema will allow defining validators that will
>>> run whenever a new value is attempted to be set on data. Messages about
>>> failed validation will be available so that the UI can display it. This
>>> system seems to be very similar to dynamic UI’s.
>>>
>>> What this all boils down to is that all of the logic required to ensure
>>> JSON’s integrity is rolled into Barricade instead of sprinkled into all
>>>of
>>> the UI code. This way, UI components can be confident in the data that
>>> they are working with, which makes their code more concise and faster to
>>> develop.
>>>
>>> Barricade seems like a great fit for Merlin because the projects it
>>> targets (Heat, Solum, Mistral, Murano) use YAML files that can be used
>>> with Barricade once they are converted to JSON.
>>>
>>>
>>> About the template builder (see screenshots attached):
>>>
>>> - Uses an interactive canvas (powered by a D3.js force-directed graph)
>>>to
>>> display Heat resources and the dependencies in between them. New
>>>resources
>>> can be drag-and-dropped from a panel onto the canvas. Different resource
>>> types are attracted to different respective areas so that they
>>> automatically arrange themselves in a familiar network topology
>>>hierarchy.
>>>  A form/panel is displayed when a resource on the canvas is clicked so
>>> that the resource’s properties can be edited.
>>>
>>> - Has two―way data binding provided by Knockout.js interfacing with
>>> Barricade.js so that editing values in forms automatically updates the
>>> topology on the canvas (e.g. Changing resource IDs or creating/removing
>>> dependencies between resources).
>>>
>>> - Allows for direct-editing of the template (via text input) and loading
>>> templates via URLs.
>>>
>>>
>>> Please, let me know if you would like a demo of the template builder and
>>> Barricade!
>>>
>>> Thanks,
>>> Drago Rosson
>>>
>>>
>>>
>>>
>>> On 8/18/14, 5:19 AM, "Timur Sufiev" <tsufiev at mirantis.com> wrote:
>>>
>>>>David,
>>>>
>>>>I'm happy to hear that :)! After thinking a bit, I came up with the
>>>>following strategy for further Merlin development: make all the
>>>>commits into a separate repository (stackforge/merlin) at least until
>>>>the PoC is ready. This will allow to keep project history more
>>>>granular instead of updating one large commit inside openstack/horizon
>>>>gerrit (thus also lessening the burden on Horizon reviewers). Once the
>>>>Merlin proceeds from the experimental/PoC phase to the implementing of
>>>>a more elaborated spec, it will be just the time for it to join with
>>>>the Horizon.
>>>>
>>>>On Wed, Aug 13, 2014 at 2:48 AM, Lyle, David <david.lyle at hp.com> wrote:
>>>>> On 8/6/14, 1:41 PM, "Timur Sufiev" <tsufiev at mirantis.com> wrote:
>>>>>
>>>>>>Hi, folks!
>>>>>>
>>>>>>Two months ago there was an announcement in ML about gathering the
>>>>>>requirements for cross-project UI library for
>>>>>>Heat/Mistral/Murano/Solum [1]. The positive feedback in related
>>>>>>googledoc [2] and some IRC chats and emails that followed convinced me
>>>>>>that I'm not the only person interested in it :), so I'm happy to make
>>>>>>the next announcement.
>>>>>>
>>>>>>The project finally has got its name - 'Merlin' (making complex UIs is
>>>>>>a kind of magic), Openstack wiki page [3] and all other stuff like
>>>>>>stackforge repo, launchpad page and IRC channel (they are all
>>>>>>referenced in [3]). For those who don't like clicking the links, here
>>>>>>is quick summary.
>>>>>>
>>>>>>Merlin aims to provide a convenient client side framework for building
>>>>>>rich UIs for Openstack projects dealing with complex input data with
>>>>>>lot of dependencies and constraints (usually encoded in YAML format
>>>>>>via some DSL) - projects like Heat, Murano, Mistral or Solum. The
>>>>>>ultimate goal for such UI is to save users from reading comprehensive
>>>>>>documentation just in order to provide correct input data, thus making
>>>>>>the UI of these projects more user-friendly. If things go well for
>>>>>>Merlin, it could be eventually merged into Horizon library (I¹ll spare
>>>>>>another option for the end of this letter).
>>>>>>
>>>>>>The framework trying to solve this ambitious task is facing at least 2
>>>>>>challenges:
>>>>>>(1) enabling the proper UX patterns and
>>>>>>(2) dealing with complexities of different projects' DSLs.
>>>>>>
>>>>>>Having worked on DSL things in Murano project before, I'm planning at
>>>>>>first to deal with the challenge (2) in the upcoming Merlin PoC. So,
>>>>>>here is the initial plan: design an in-framework object model (OM)
>>>>>>that could translated forth and back into target project's DSL. This
>>>>>>OM is meant to be synchronised with visual elements shown on browser
>>>>>>canvas. Target project is the Heat with its HOT templates - it has the
>>>>>>most well-established syntax among other projects and comprehensive
>>>>>>documentation.
>>>>>>
>>>>>>Considering the challenge (1), not being a dedicated UX engineer, I'm
>>>>>>planning to start with some rough UI concepts [4] and gradually
>>>>>>improve them relying on community feedback, and especially, Openstack
>>>>>>UX group. If anybody from the UX team (or any other team!) is willing
>>>>>>to be involved to a greater degree than just giving some feedback,
>>>>>>you're are enormously welcome! Join Merlin, it will be fun :)!
>>>>>>
>>>>>>Finally, with this announcement I¹d like to start a discussion with
>>>>>>Horizon community. As far as I know, Horizon in its current state
>>>>>>lacks such UI toolkit as Merlin aims to provide. Would it be by any
>>>>>>chance possible for the Merlin project to be developed from the very
>>>>>>beginning as part of Horizon library? This choice has its pros and
>>>>>>cons I¹m aware of, but I¹d like to hear the opinions of Horizon
>>>>>>developers on that matter.
>>>>>
>>>>> I would like to see this toolset built into Horizon. That will make it
>>>>> accessible to integrated projects like Heat that Horizon already
>>>>>supports,
>>>>> but will also allow other projects to use the horizon library as a
>>>>> building block to providing managing project specific DSLs.
>>>>>
>>>>> David
>>>>>
>>>>>>
>>>>>>[1]
>>>>>>http://lists.openstack.org/pipermail/openstack-dev/2014-June/037054.ht
>>>>>>ml
>>>>>>[2]
>>>>>>https://docs.google.com/a/mirantis.com/document/d/19Q9JwoO77724RyOp7Xk
>>>>>>pY
>>>>>>mA
>>>>>>Lwmdb7JjoQHcDv4ffZ-I/edit#
>>>>>>[3] https://wiki.openstack.org/wiki/Merlin
>>>>>>[4] https://wiki.openstack.org/wiki/Merlin/SampleUI
>>>>>>
>>>>>>--
>>>>>>Timur Sufiev
>>>>>>
>>>>>>_______________________________________________
>>>>>>OpenStack-dev mailing list
>>>>>>OpenStack-dev at lists.openstack.org
>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>>--
>>>>Timur Sufiev
>>>>
>>>>_______________________________________________
>>>>OpenStack-dev mailing list
>>>>OpenStack-dev at lists.openstack.org
>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>>--
>>Timur Sufiev
>>
>>_______________________________________________
>>OpenStack-dev mailing list
>>OpenStack-dev at lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Timur Sufiev