[openstack-dev] [Mistral] Changing "expression" delimiters in Mistral DSL
Renat Akhmerov
rakhmerov at mirantis.com
Thu Feb 19 06:24:14 UTC 2015
Guys,
I really appreciate the input of you all.
We decided that ideally we need to agree on that syntax within days, not weeks or months. But anyway, since we started this discussion yesterday I just want to give us extra 1-2 days to play with all these thoughts in our heads.
Just one additional maybe a little bit crazy idea to the pile we’ve already made:
What if we officially allow more than one delimiter syntax? Why stick to just one? Technically I’m 99% sure it’s doable. As long as it’s clearly pointed in the documentation there shouldn’t problems with understanding. Just one con (IMO, little) that I see here is “if so then what syntax do we use to write all our examples, docs, tutorials etc. etc.”
So obviously one of the options (just for majority of votes) would be <% %> and something else less familiar but more smoother looking in YAML such as yaql{…} or <{…}>.
I’m ready to be beaten by stones! Fire away :)
Renat Akhmerov
@ Mirantis Inc.
> On 19 Feb 2015, at 11:25, James Fryman <james at stackstorm.com> wrote:
>
> On Feb 18, 2015, at 3:07 PM, Dmitri Zimine <dzimine at stackstorm.com <mailto:dzimine at stackstorm.com>> wrote:
>
>>> Syntax options that we’d like to discuss further
>>>
>>> <% 1 + 1 %> # pro- ruby/js/puppet/chef familiarity con - spaces, and % is too large symbol
>>> <{1 + 1}> # pro - less spaces, con - no familiarity
>>> <? 1 + 1 ?> # php familiarity, need spaces
>>>
>>> The primary criteria to select these 3 options is that they are YAML compatible. Technically they all would solve our problems (primarily no embracing quotes needed like in Ansible so no ambiguity on data types).
>>>
>>> The secondary criteria is syntax symmetry. After all I agree with Patrick's point about better readability when we have opening and closing sequences alike.
>>
>>
>> To me, another critical criteria is familiarity: target users - openstack developers and devops, familiar with the delimiters.
>>
>> That is why of the three above I prefer <% %> .
>>
>> It is commonly used in Puppet/Chef [1], Ruby, Javascript. One won’t be surprised to see it and won’t need to change the muscle memory to type open/closed characters especially when working on say Puppet and Mistral at the same time (not unlikely).
>>
>>
>> [1] https://docs.puppetlabs.com/guides/templating.html#erb-is-plain-text-with-embedded-ruby <https://docs.puppetlabs.com/guides/templating.html#erb-is-plain-text-with-embedded-ruby>
>>
>
>
> I have been lurking on this thread, and just wanted to toss in $0.02 as you all deliberate. In truth, any of the options Renat highlights would be fine, and the points made to arrive at the final choices are sound. The end result will types will be explicit, and that is great. In light of this though, using the <% %> syntax is still ideal if only for one reason: friction.
>
> In a recent discussion with a colleague of mine, he told me that in his daily job, he is so busy and slammed with operations tasks, his measure of a tool he will use is whether it provides value within 30-60 minutes. Otherwise, there is a fire somewhere else that needs to be put out and cannot be bothered.
>
> To be frank, there is no way that this proposed syntax change and how it is ultimately decorated is going to be a game changer to how future users will evaluate Mistral. But, at 3am <x-apple-data-detectors://4> in the morning during a production outage where an Ops admin is hotpatching a workflow to get things moving again... That disparity does in fact matter. Eyes are already trained to look for <% %>, the large amount of spacing draws the eyes, and it's a known function in other existing ops toolkits. Less friction to adoption/less friction to troubleshoot.I think these are all pluses. I want to be drawn to where things are changing or dynamic in my templates to aid in troubleshooting.
>
> Again, this is just another data point to throw out. While users are busy trying to absorb all that is workflow creation/design... Having as many likenesses and anchors to existing tools can certainly not hurt adoption.
>
> Thank you all for your efforts!
>
>>
>> On Feb 18, 2015, at 3:20 AM, Renat Akhmerov <rakhmerov at mirantis.com <mailto:rakhmerov at mirantis.com>> wrote:
>>
>>> Hi again,
>>>
>>> Sorry, I started writing this email before Angus replied so I will shoot it as is and then we can continue…
>>>
>>>
>>> So after discussing all the options again with a small group of team members we came to the following things:
>>>
>>> Syntax options that we’d like to discuss further
>>>
>>> <% 1 + 1 %> # pro- ruby/js/puppet/chef familiarity con - spaces, and % is too large symbol
>>> <{1 + 1}> # pro - less spaces, con - no familiarity
>>> <? 1 + 1 ?> # php familiarity, need spaces
>>>
>>> The primary criteria to select these 3 options is that they are YAML compatible. Technically they all would solve our problems (primarily no embracing quotes needed like in Ansible so no ambiguity on data types).
>>>
>>> The secondary criteria is syntax symmetry. After all I agree with Patrick's point about better readability when we have opening and closing sequences alike.
>>>
>>> Some additional details can be found in [0]
>>>
>>>
>>> [0] https://etherpad.openstack.org/p/mistral-YAQL-delimiters <https://etherpad.openstack.org/p/mistral-YAQL-delimiters>
>>>
>>> Renat Akhmerov
>>> @ Mirantis Inc.
>>>
>>>
>>>> On 18 Feb 2015, at 07:37, Patrick Hoolboom <patrick at stackstorm.com <mailto:patrick at stackstorm.com>> wrote:
>>>>
>>>> My main concern with the {} delimiters in YAQL is that the curly brace already has a defined use within YAML. We most definitely will eventually run in to parsing errors with whatever delimiter we choose but I don't feel that it should conflict with the markup language it is directly embedded in. It gets quite difficult to, at a glance, identify YAQL expressions. <% %> may appear ugly to some but I feel that it works as a clear delimiter of both the beginning AND the end of the YAQL query. The options that only escape the beginning look fine in small examples like this but the workflows that we have written or seen in the wild tend to have some fairly large expressions. If the opening and closing delimiters don't match, it gets quite difficult to read.
>>>>
>>>>> From: Anastasia Kuznetsova <akuznetsova at mirantis.com <mailto:akuznetsova at mirantis.com>>
>>>>> Subject: Re: [openstack-dev] [Mistral] Changing "expression" delimiters in Mistral DSL
>>>>> Date: February 17, 2015 at 8:28:27 AM PST
>>>>> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org <mailto:openstack-dev at lists.openstack.org>>
>>>>> Reply-To: "OpenStack Development Mailing List \(not for usage questions\)" <openstack-dev at lists.openstack.org <mailto:openstack-dev at lists.openstack.org>>
>>>>>
>>>>> As for me, I think that <% ... %> is not an elegant solution and looks massive because of '%' sign. Also I agree with Renat, that <% ... %> reminds HTML/Jinja2 syntax.
>>>>>
>>>>> I am not sure that similarity with something should be one of the main criteria, because we don't know who will use Mistral.
>>>>>
>>>>> I like:
>>>>> - <{1 + $.var}> Renat's example
>>>>> - variant with using some functions (item 2 in Dmitry's list): { yaql: “1+1+$.my.var < 100” } or <yaql: 'Hello' + $.name >
>>>>> - my two cents, maybe we can use something like: result: -< "Hello" + $.name >-
>>>>>
>>>>>
>>>>> Regards,
>>>>> Anastasia Kuznetsova
>>>>>
>>>>> On Tue, Feb 17, 2015 at 1:17 PM, Nikolay Makhotkin <nmakhotkin at mirantis.com <mailto:nmakhotkin at mirantis.com>> wrote:
>>>>> Some suggestions from me:
>>>>>
>>>>> 1. <y 1 + $.var > # (short from yaql).
>>>>> 2. <{ 1 + $.var }> # as for me, looks more elegant than <% %>. And visually it is more strong
>>>>>
>>>>> A also like p7 and p8 suggested by Renat.
>>>>>
>>>>> On Tue, Feb 17, 2015 at 11:43 AM, Renat Akhmerov <rakhmerov at mirantis.com <mailto:rakhmerov at mirantis.com>> wrote:
>>>>> One more:
>>>>>
>>>>> p9: \{1 + $.var} # That’s pretty much what https://review.openstack.org/#/c/155348/ <https://review.openstack.org/#/c/155348/> addresses but it’s not exactly that. Note that we don’t have to put it in quotes in this case to deal with YAML {} semantics, it’s just a string
>>>>>
>>>>>
>>>>>
>>>>> Renat Akhmerov
>>>>> @ Mirantis Inc.
>>>>>
>>>>>
>>>>>
>>>>>> On 17 Feb 2015, at 13:37, Renat Akhmerov <rakhmerov at mirantis.com <mailto:rakhmerov at mirantis.com>> wrote:
>>>>>>
>>>>>> Along with <% %> syntax here are some other alternatives that I checked for YAML friendliness with my short comments:
>>>>>>
>>>>>> p1: ${1 + $.var} # Here it’s bad that $ sign is used for two different things
>>>>>> p2: ~{1 + $.var} # ~ is easy to miss in a text
>>>>>> p3: ^{1 + $.var} # For someone may be associated with regular expressions
>>>>>> p4: ?{1 + $.var}
>>>>>> p5: <{1 + $.var}> # This is kinda crazy
>>>>>> p6: e{1 + $.var} # That looks a pretty interesting option to me, “e” could mean “expression” here.
>>>>>> p7: yaql{1 + $.var} # This is interesting because it would give a clear and easy mechanism to plug in other expression languages, “yaql” here is a used dialect for the following expression
>>>>>> p8: y{1 + $.var} # “y” here is just shortened “yaql"
>>>>>>
>>>>>>
>>>>>> Any ideas and thoughts would be really appreciated!
>>>>>>
>>>>>> Renat Akhmerov
>>>>>> @ Mirantis Inc.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 17 Feb 2015, at 12:53, Renat Akhmerov <rakhmerov at mirantis.com <mailto:rakhmerov at mirantis.com>> wrote:
>>>>>>>
>>>>>>> Dmitri,
>>>>>>>
>>>>>>> I agree with all your reasonings and fully support the idea of changing the syntax now as well as changing system’s API a little bit due to recently found issues in the current engine design that don’t allow us, for example, to fully implement ‘with-items’ (although that’s a little bit different story).
>>>>>>>
>>>>>>> Just a general note about all changes happening now: Once we release kilo stable release our API, DSL of version 2 must be 100% stable. I was hoping to stabilize it much earlier but the start of production use revealed a number of things (I think this is normal) which we need to address, but not later than the end of Kilo.
>>>>>>>
>>>>>>> As far as <% %> syntax. I see that it would solve a number of problems (YAML friendliness, type ambiguity) but my only not strong argument is that it doesn’t look that elegant in YAML as it looks, for example, in ERB templates. It really reminds me XML/HTML and looks like a bear in a grocery store (tried to make it close to one old russian saying :) ). So just for this only reason I’d suggest we think about other alternatives, maybe not so familiar to Ruby/Chef/Puppet users but looking better with YAML and at the same time being YAML friendly.
>>>>>>>
>>>>>>> I would be good if we could here more feedback on this, especially from people who started using Mistral.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Renat Akhmerov
>>>>>>> @ Mirantis Inc.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 17 Feb 2015, at 03:06, Dmitri Zimine <dzimine at stackstorm.com <mailto:dzimine at stackstorm.com>> wrote:
>>>>>>>>
>>>>>>>> SUMMARY:
>>>>>>>> ----------------
>>>>>>>>
>>>>>>>> We are changing the syntax for inlining YAQL expressions in Mistral YAML from {1+$.my.var} (or “{1+$.my.var}”) to <% 1+$.my.var %>
>>>>>>>>
>>>>>>>> Below I explain the rationale and the criteria for the choice. Comments and suggestions welcome.
>>>>>>>>
>>>>>>>> DETAILS:
>>>>>>>> -------------
>>>>>>>>
>>>>>>>> We faced a number of problems with using YAQL expressions in Mistral DSL: [1] must handle any YAQL, not only the ones started with $; [2] must preserve types and [3] must comply with YAML. We fixed these problems by applying Ansible style syntax, requiring quotes around delimiters (e.g. “{1+$.my.yaql.var}”). However, it lead to unbearable confusion in DSL readability, in regards to types:
>>>>>>>>
>>>>>>>> publish:
>>>>>>>> intvalue1: "{1+1}” # Confusing: you expect quotes to be string.
>>>>>>>> intvalue2: "{int(1+1)}” # Even this doestn’ clean the confusion
>>>>>>>> whatisthis:"{$.x + $.y}” # What type would this return?
>>>>>>>>
>>>>>>>> We got a very strong push back from users in the filed on this syntax.
>>>>>>>>
>>>>>>>> The crux of the problem is using { } as delimiters YAML. It is plain wrong to use the reserved character. The clean solution is to find a delimiter that won’t conflict with YAML.
>>>>>>>>
>>>>>>>> Criteria for selecting best alternative are:
>>>>>>>> 1) Consistently applies to to all cases of using YAML in DSL
>>>>>>>> 2) Complies with YAML
>>>>>>>> 3) Familiar to target user audience - openstack and devops
>>>>>>>>
>>>>>>>> We prefer using two-char delimiters to avoid requiring extra escaping within the expressions.
>>>>>>>>
>>>>>>>> The current winner is <% %>. It fits YAML well. It is familiar to openstack/devops as this is used for embedding Ruby expressions in Puppet and Chef (for instance, [4]). It plays relatively well across all cases of using expressions in Mistral (see examples in [5]):
>>>>>>>>
>>>>>>>> ALTERNATIVES considered:
>>>>>>>> --------------------------------------------------
>>>>>>>>
>>>>>>>> 1) Use Ansible-like syntax: http://docs.ansible.com/YAMLSyntax.html#gotchas <http://docs.ansible.com/YAMLSyntax.html#gotchas>
>>>>>>>> Rejected for confusion around types. See above.
>>>>>>>>
>>>>>>>> 2) Use functions, like Heat HOT or TOSCA:
>>>>>>>>
>>>>>>>> HOT templates and TOSCA doesn’t seem to have a concept of typed variables to borrow from (please correct me if I missed it). But they have functions: function: { function_name: {foo: [parameter1, parameter 2], bar:"xxx”}}. Applied to Mistral, it would look like:
>>>>>>>>
>>>>>>>> publish:
>>>>>>>> - bool_var: { yaql: “1+1+$.my.var < 100” }
>>>>>>>>
>>>>>>>> Not bad, but currently rejected as it reads worse than delimiter-based syntax, especially in simplified one-line action invocation.
>>>>>>>>
>>>>>>>> 3) < > paired with other symbols: php-styoe <? ..?>
>>>>>>>>
>>>>>>>>
>>>>>>>> REFERENCES:
>>>>>>>> ----------------------
>>>>>>>>
>>>>>>>> [1] Allow arbitrary YAQL expressions, not just ones started with $ : https://github.com/stackforge/mistral/commit/5c10fb4b773cd60d81ed93aec33345c0bf8f58fd <https://github.com/stackforge/mistral/commit/5c10fb4b773cd60d81ed93aec33345c0bf8f58fd>
>>>>>>>> [2] Use Ansible-like syntax to make YAQL expressions YAML complient https://github.com/stackforge/mistral/commit/d9517333b1fc9697d4847df33d3b774f881a111b <https://github.com/stackforge/mistral/commit/d9517333b1fc9697d4847df33d3b774f881a111b>
>>>>>>>> [3] Preserving types in YAQL https://github.com/stackforge/mistral/blob/d9517333b1fc9697d4847df33d3b774f881a111b/mistral/tests/unit/test_expressions.py#L152-L184 <https://github.com/stackforge/mistral/blob/d9517333b1fc9697d4847df33d3b774f881a111b/mistral/tests/unit/test_expressions.py#L152-L184>
>>>>>>>> [4]Using <% %> in Puppet https://docs.puppetlabs.com/guides/templating.html#erb-is-plain-text-with-embedded-ruby <https://docs.puppetlabs.com/guides/templating.html#erb-is-plain-text-with-embedded-ruby>
>>>>>>>> [5] Etherpad with discussion https://etherpad.openstack.org/p/mistral-YAQL-delimiters <https://etherpad.openstack.org/p/mistral-YAQL-delimiters>
>>>>>>>> [6] Blueprint https://blueprints.launchpad.net/mistral/+spec/yaql-delimiters <https://blueprints.launchpad.net/mistral/+spec/yaql-delimiters>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe <http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Nikolay
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe <http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>>>
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org <mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org <mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org <mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150219/19e9f188/attachment.html>
More information about the OpenStack-dev
mailing list