Open Stack

Timur,

Composable entities can be a real need for Heat if provider templates
(which allow templates to be used as a resource, with a template’s
parameters and outputs becoming properties and attributes, respectively)
are to be included in the app. A provider template resource, since it is a
template itself, would be composed of resources which would require a
composable entity. What is great about D3’s force graph is that it’s nodes
and links can be completely arbitrary - meaning they can be any JavaScript
object (including an SVG or DOM element). Additionally, the force graph
simulation updates x and y properties on those elements and calls a
user-defined “tick” function. The tick function can use the x and y
properties in any way it wants to do the *actual* update to the position
of each element. For example, this is how multiple foci can be implemented
[1]. Lots of other customization is available, including starting and
stopping the simulation, updating the node and link data, and having
per-element control of most (all?) properties such as charge or link
distance.

Composability can be achieved using SVG’s <g> elements to group multiple
graphical elements together. The tick function would need to update the
<g>’s transform attribute [2]. This is how it is done in my app since my
nodes and links are composed of icons, labels, backgrounds, etc. I think
that D3’s force graph is not a limiting factor since it itself does not
concern itself with graphics at all. Therefore, the question seems to be
whether D3 can do everything graphically that Merlin needs. D3 is not a
graphics API, but it does have support for graphical manipulation,
animations, and events. They have sufficed for me so far. Plus, D3 can do
these things without having to use its fancy data transformations so it
can be used as a low-level SVG library where necessary. D3 can do a lot
[3] so hopefully it could also do what Merlin needs.

You are in luck, because I have just now open-sourced Barricade! Check it
out [4]. I am working on getting documentation written for it but to see
some ways it can be used, look at its test suite [5].

[1] http://bl.ocks.org/mbostock/1021953
[2] node.attr("transform", function (d) {
        return "translate(" + d.x + ", " + d.y + ")";
    });
[3] http://christopheviau.com/d3list/
[4] https://github.com/rackerlabs/barricade

[5] 
https://github.com/rackerlabs/barricade/blob/master/test/barricade_Spec.js

On 8/28/14, 10:03 AM, "Timur Sufiev" <tsufiev at mirantis.com> wrote:

>Hello, Drago!
>
>I'm extremely interested in learning more about your HOT graphical
>builder. The screenshots you had attached look gorgeous! Yours visual
>representation of Heat resources is much more concise and simple than
>I had drawn in Merlin PoC mock-ups [1]. On the other hand I have some
>suspicions that D3.js is a good fit for a general purpose UI toolkit
>Merlin aims to provide. Please don't get me wrong, D3.js is a great
>library which can do fantastic things with data - in case your
>data<->visualization use-case maps to the one of the facilities D3.js
>provides out of the box. In case it doesn't, there are 2 options:
>either change your approach to what should be visualized/how it should
>be visualized, or tweak some inner machinery of D3.js
>
>While bending the design towards the facilities of D3.js doesn't seem
>a viable choice, changing D3.js from inside can be painful too. AFAIK
>force-directed graph layout from D3.js doesn't provide the means to
>represent composable entities (which isn't a big problem for Heat, but
>is a very serious issue for Murano) out of the box. By composable I
>mean something like [2] - but with much more complex inner structure
>(imagine the Resource entity [3] having as its properties other
>Resource entities which are shown as simple rounded rectangles with
>labels on that picture, but are expanded into complex objects similar
>to [3] once the user, say, clicks on them). As far as I understand,
>you are visualizing that kind of composition via arrow links, but I'd
>like to try another design options (especially in case of Murano) and
>fear that D3.js will constrain me here. I've been thinking a bit about
>using more low-level SVG js-framework like Raphael.js - it doesn't
>offer most of the goodies D3.js does, but also it doesn't force me to
>create the design based on some data transformations in a way that
>D3.js does, providing the good old procedural API instead. Of course,
>I may be wrong, perhaps more time and efforts invested into Merlin PoC
>would allow me to realize it (or not).

>Yet you are totally right having stressed the importance of right tool
>for implementing the underlying object model (or JSON-wrapper as you
>called it) - Barricade.js. That's the second big part of work Merlin
>had to do, and I couldn't underestimate how it would be beneficial for
>Merlin to leverage some of the facilities that Barricade.js provides.
>I'll gladly look at the demo of template builder and Barricade. Is
>there any chance I could take a look also at the source code of
>Barricade.js, so I would better understand to which extent it suits
>Merlin’s needs? I've searched through github.com and didn't found any
>traces of Barricade.js repo, so it seems like some in-house project to
>me. What are your plans for sharing this library with the community?
>
>[1] https://wiki.openstack.org/wiki/Merlin/PoC
>[2] http://mbostock.github.io/d3/talk/20111116/pack-hierarchy.html
>[3] https://wiki.openstack.org/wiki/File:Merlin_poc_3.png
>
>On Wed, Aug 27, 2014 at 6:14 PM, Drago Rosson
><drago.rosson at rackspace.com> wrote:
>> Hello Timur,
>>
>> I have been developing a graphical Heat Orchestration Template builder.
>>I
>> first heard about the Merlin project around a week ago and found that
>>what
>> I have developed in the past few months is in line with what Merlin is
>> trying to accomplish.
>>
>> Some of the features of the template builder (described further down)
>> could be used as part of the UI framework Merlin aims to produce.
>>However,
>> its most important and useful component is a JavaScript library I have
>> developed called Barricade.js, which powers the template builder’s data
>> layer.
>>
>> Barricade.js aims to solve the problem of using JSON data across a web
>> app. It creates data model objects out of JSON using a predefined schema
>> (which looks similar to the schema used in your Murano Dynamic UI). This
>> removes the disadvantages with JSON and introduces some very important
>> advantages:
>>
>> - Encapsulation. JSON values can be set to any type or deleted entirely,
>> which either causes errors when UI components expect these values to
>>exist
>> or be of a certain type, or forces the UI components to constantly check
>> for correctness. Barricade instead wraps around the JSON and provides
>> accessor methods to ensure type-safe data manipulation. Additionally,
>> Barricade objects are observable, so changes made to their data trigger
>> events that can be subscribed to by UI components.
>>
>> - Normalization. Whenever properties that are defined in the schema are
>> missing in the JSON, Barricade will fill them in with default values.
>>This
>> way, UIs will always have valid values where they expect them, making
>> their design much simpler. Optional properties are extremely common in
>> Heat templates.
>>
>> - Metadata. Anything extra attached to JSON must be handled carefully
>> (such as when converting back to the original YAML format). By wrapping
>> the JSON with Barricade, metadata and convenience methods that UI
>> components can use can be defined. For instance, the datatype of any
>>value
>> or a description to go along with each property in a Heat resource.
>>
>> - Validation. Soon, the schema will allow defining validators that will
>> run whenever a new value is attempted to be set on data. Messages about
>> failed validation will be available so that the UI can display it. This
>> system seems to be very similar to dynamic UI’s.
>>
>> What this all boils down to is that all of the logic required to ensure
>> JSON’s integrity is rolled into Barricade instead of sprinkled into all
>>of
>> the UI code. This way, UI components can be confident in the data that
>> they are working with, which makes their code more concise and faster to
>> develop.
>>
>> Barricade seems like a great fit for Merlin because the projects it
>> targets (Heat, Solum, Mistral, Murano) use YAML files that can be used
>> with Barricade once they are converted to JSON.
>>
>>
>> About the template builder (see screenshots attached):
>>
>> - Uses an interactive canvas (powered by a D3.js force-directed graph)
>>to
>> display Heat resources and the dependencies in between them. New
>>resources
>> can be drag-and-dropped from a panel onto the canvas. Different resource
>> types are attracted to different respective areas so that they
>> automatically arrange themselves in a familiar network topology
>>hierarchy.
>>  A form/panel is displayed when a resource on the canvas is clicked so
>> that the resource’s properties can be edited.
>>
>> - Has two―way data binding provided by Knockout.js interfacing with
>> Barricade.js so that editing values in forms automatically updates the
>> topology on the canvas (e.g. Changing resource IDs or creating/removing
>> dependencies between resources).
>>
>> - Allows for direct-editing of the template (via text input) and loading
>> templates via URLs.
>>
>>
>> Please, let me know if you would like a demo of the template builder and
>> Barricade!
>>
>> Thanks,
>> Drago Rosson
>>
>>
>>
>>
>> On 8/18/14, 5:19 AM, "Timur Sufiev" <tsufiev at mirantis.com> wrote:
>>
>>>David,
>>>
>>>I'm happy to hear that :)! After thinking a bit, I came up with the
>>>following strategy for further Merlin development: make all the
>>>commits into a separate repository (stackforge/merlin) at least until
>>>the PoC is ready. This will allow to keep project history more
>>>granular instead of updating one large commit inside openstack/horizon
>>>gerrit (thus also lessening the burden on Horizon reviewers). Once the
>>>Merlin proceeds from the experimental/PoC phase to the implementing of
>>>a more elaborated spec, it will be just the time for it to join with
>>>the Horizon.
>>>
>>>On Wed, Aug 13, 2014 at 2:48 AM, Lyle, David <david.lyle at hp.com> wrote:
>>>> On 8/6/14, 1:41 PM, "Timur Sufiev" <tsufiev at mirantis.com> wrote:
>>>>
>>>>>Hi, folks!
>>>>>
>>>>>Two months ago there was an announcement in ML about gathering the
>>>>>requirements for cross-project UI library for
>>>>>Heat/Mistral/Murano/Solum [1]. The positive feedback in related
>>>>>googledoc [2] and some IRC chats and emails that followed convinced me
>>>>>that I'm not the only person interested in it :), so I'm happy to make
>>>>>the next announcement.
>>>>>
>>>>>The project finally has got its name - 'Merlin' (making complex UIs is
>>>>>a kind of magic), Openstack wiki page [3] and all other stuff like
>>>>>stackforge repo, launchpad page and IRC channel (they are all
>>>>>referenced in [3]). For those who don't like clicking the links, here
>>>>>is quick summary.
>>>>>
>>>>>Merlin aims to provide a convenient client side framework for building
>>>>>rich UIs for Openstack projects dealing with complex input data with
>>>>>lot of dependencies and constraints (usually encoded in YAML format
>>>>>via some DSL) - projects like Heat, Murano, Mistral or Solum. The
>>>>>ultimate goal for such UI is to save users from reading comprehensive
>>>>>documentation just in order to provide correct input data, thus making
>>>>>the UI of these projects more user-friendly. If things go well for
>>>>>Merlin, it could be eventually merged into Horizon library (I¹ll spare
>>>>>another option for the end of this letter).
>>>>>
>>>>>The framework trying to solve this ambitious task is facing at least 2
>>>>>challenges:
>>>>>(1) enabling the proper UX patterns and
>>>>>(2) dealing with complexities of different projects' DSLs.
>>>>>
>>>>>Having worked on DSL things in Murano project before, I'm planning at
>>>>>first to deal with the challenge (2) in the upcoming Merlin PoC. So,
>>>>>here is the initial plan: design an in-framework object model (OM)
>>>>>that could translated forth and back into target project's DSL. This
>>>>>OM is meant to be synchronised with visual elements shown on browser
>>>>>canvas. Target project is the Heat with its HOT templates - it has the
>>>>>most well-established syntax among other projects and comprehensive
>>>>>documentation.
>>>>>
>>>>>Considering the challenge (1), not being a dedicated UX engineer, I'm
>>>>>planning to start with some rough UI concepts [4] and gradually
>>>>>improve them relying on community feedback, and especially, Openstack
>>>>>UX group. If anybody from the UX team (or any other team!) is willing
>>>>>to be involved to a greater degree than just giving some feedback,
>>>>>you're are enormously welcome! Join Merlin, it will be fun :)!
>>>>>
>>>>>Finally, with this announcement I¹d like to start a discussion with
>>>>>Horizon community. As far as I know, Horizon in its current state
>>>>>lacks such UI toolkit as Merlin aims to provide. Would it be by any
>>>>>chance possible for the Merlin project to be developed from the very
>>>>>beginning as part of Horizon library? This choice has its pros and
>>>>>cons I¹m aware of, but I¹d like to hear the opinions of Horizon
>>>>>developers on that matter.
>>>>
>>>> I would like to see this toolset built into Horizon. That will make it
>>>> accessible to integrated projects like Heat that Horizon already
>>>>supports,
>>>> but will also allow other projects to use the horizon library as a
>>>> building block to providing managing project specific DSLs.
>>>>
>>>> David
>>>>
>>>>>
>>>>>[1]
>>>>>http://lists.openstack.org/pipermail/openstack-dev/2014-June/037054.ht
>>>>>ml
>>>>>[2]
>>>>>https://docs.google.com/a/mirantis.com/document/d/19Q9JwoO77724RyOp7Xk
>>>>>pY
>>>>>mA
>>>>>Lwmdb7JjoQHcDv4ffZ-I/edit#
>>>>>[3] https://wiki.openstack.org/wiki/Merlin
>>>>>[4] https://wiki.openstack.org/wiki/Merlin/SampleUI
>>>>>
>>>>>--
>>>>>Timur Sufiev
>>>>>
>>>>>_______________________________________________
>>>>>OpenStack-dev mailing list
>>>>>OpenStack-dev at lists.openstack.org
>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>--
>>>Timur Sufiev
>>>
>>>_______________________________________________
>>>OpenStack-dev mailing list
>>>OpenStack-dev at lists.openstack.org
>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
>-- 
>Timur Sufiev
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev