Open Stack

Renat,

Thanks for the detailed explanation. It is quite instructive and helps
frame the question accurately by providing the necessary context.

   - Is the approach itself reasonable?

[Manas] I think so. A repeatable workflow requires these sort of separation
i.e. a model in which to save the specification as supplied by the user and
a model in which to save the execution state - current and past.

   - Do we have better ideas on how to work with DSL? A good mental
   exercise here would be to imagine that we have more than one DSL, not only
   YAML but say XML. How would it change the picture?

[Manas] As long as we form an abstraction between the DSL format i.e.
YAML/XML and it consumption we will be able to move between various
formats. (wishful) My personal preference is to not even have DSL show up
anywhere in Mistral code apart from take it as input and transforming into
this first level specification model - I know this is not the current state.

   - How can we clearly distinguish between these two models so that it
   wouldn't be confusing?
   - Do we have a better naming in mind?

[Manas] I think we all would agree that the best approach is to have
precise naming.

I see your point of de-normalizing the dsl data into respective db model
objects.

In a previous email I suggested using *Spec. I will try to build on this -
1. Everything specified via the YAML input is a specification or definition
or template. Therefore I suggest we suffix all these types with
Spec/Definition/Template. So TaskSpec/TaskDefinition/TaskTemplate etc.. As
per the latest change these are TaskData ... ActionData.
2. As per current impl the YAML is stored as a key-value in the DB. This is
fine since that is front-ended by objects that Nikolay has introduced. e.g.
TaskData, ActionData etc.
3. As per my thinking a model object that ends up in the DB or a model
object that is in memory all can reside in the same module. I view
persistence as an orthogonal concern so no real reason to distinguish the
module names of the two set of models. If we do choose to distinguish as
per latest change i.e. mistral/workbook that works too.

@Nikolay - I am generally ok with the approach. I hope that this helps
clarify my thinking and perception. Please ask more questions.

Overall I like the approach of formalizing the 2 models. I am ok with
current state of the review and have laid out my preferences.

Thanks,
Manas

On Wed, Mar 5, 2014 at 3:39 AM, Nikolay Makhotkin
<nmakhotkin at mirantis.com>wrote:

> I think today and I have a good name for package (instead of
> 'mistral/model')
>
> How do you think about to name it 'mistral/workbook'? I.e., It means that
> it contains modules for work with workbook representation - tasks,
> services, actions and workflow.
>
> This way we able to get rid of any confusing.
>
>
> Best Regards,
> Nikolay
>
>
> On Wed, Mar 5, 2014 at 8:50 AM, Renat Akhmerov <rakhmerov at mirantis.com>wrote:
>
>> I think we forgot to point to the commit itself. Here it is:
>> https://review.openstack.org/#/c/77126/
>>
>> Manas, can you please provide more details on your suggestion?
>>
>> For now let me just describe the background of Nikolay's question.
>>
>> Basically, we are talking about how we are working with data inside
>> Mistral. So far, for example, if a user sent a request to Mistral "start
>> workflow" then Mistral would do the following:
>>
>>    - Get workbook DSL (YAML) from the DB (given that it's been already
>>    persisted earlier).
>>    - Load it into a dictionary-like structure using standard 'yaml'
>>    library.
>>    - Based on this dictionary-like structure create all necessary DB
>>    objects to track the state of workflow execution objects and individual
>>    tasks.
>>    - Perform all the necessary logic in engine and so on. The important
>>    thing here is that DB objects contain corresponding DSL snippets as they
>>    are described in DSL (e.g. tasks have property "task_dsl") to reduce the
>>    complexity of relational model that we have in DB. Otherwise it would be
>>    really complicated and most of the queries would contain lots of joins. The
>>    example of non-trivial relation in DSL is "task"->"action
>>    name"->"service"->"service actions"->"action", as you can see it would be
>>    hard to navigate to action in the DB from a task if our relational model
>>    matches to what we have in DSL. this approach leads to the problem of
>>    addressing any dsl properties using hardcoded strings which are spread
>>    across the code and that brings lots of pain when doing refactoring, when
>>    trying to understand the structure of the model we describe in DSL, it
>>    doesn't allow to do validation easily and so on.
>>
>>
>> So what we have in DB we've called "model" so far and we've called just
>> "dsl" the dictionary structure coming from DSL. So if we got a part of the
>> structure related to a task we would call it "dsl_task".
>>
>> So what Nikolay is doing now is he's reworking the approach how we work
>> with DSL. Now we assume that after we parsed a workbook DSL we get some
>> "model". So that we don't use "dsl" in the code anywhere this "model"
>> describes basically the structure of what we have in DSL and that would
>> allow to address the problems I mentioned above (hardcoded strings are
>> replaced with access methods, we clearly see the structure of what we're
>> working with, we can easily validate it and so on). So when we need to
>> access some DSL properties we would need to get workbook DSL from DB, build
>> this model out of it and continue to work with it.
>>
>> Long story short, this model parsed from DSL is not the model we store in
>> DB but they're both called "model" which may be confusing. For me this
>> non-DB model more looks like "domain model" or something like this. So the
>> question I would ask ourselves here:
>>
>>    - Is the approach itself reasonable?
>>    - Do we have better ideas on how to work with DSL? A good mental
>>    exercise here would be to imagine that we have more than one DSL, not only
>>    YAML but say XML. How would it change the picture?
>>    - How can we clearly distinguish between these two models so that it
>>    wouldn't be confusing?
>>    - Do we have a better naming in mind?
>>
>>
>> Thanks.
>>
>>  Renat Akhmerov
>> @ Mirantis Inc.
>>
>>
>>
>> On 05 Mar 2014, at 08:56, Manas Kelshikar <manas at stackstorm.com> wrote:
>>
>> Since the renaming is for types in mistral.model.*. I am thinking we
>> suffix with Spec e.g.
>>
>> TaskObject -> TaskSpec
>> ActionObject -> ActionSpec and so on.
>>
>> The "Spec" suggest that it is a specification of the final object that
>> ends up in the DB and not the actual object. Multiple actual objects can be
>> derived from these Spec objects which fits well with the current paradigm.
>> Thoughts?
>>
>>
>> On Mon, Mar 3, 2014 at 9:43 PM, Manas Kelshikar <manas at stackstorm.com>wrote:
>>
>>> Hi Nikolay -
>>>
>>> Is your concern that mistral.db.sqlalchemy.models.* and mistral.model.*
>>> will lead to confusion or something else?
>>>
>>> IMHO as per your change model seems like the appropriate usage while
>>> what is stored in the DB is also a model. If we pick appropriate names to
>>> distinguish between nature of the objects we should be able to avoid any
>>> confusion and whether or not model appears in the module name should not
>>> matter much.
>>>
>>> Thanks,
>>> Manas
>>>
>>>
>>> On Mon, Mar 3, 2014 at 8:43 AM, Nikolay Makhotkin <
>>> nmakhotkin at mirantis.com> wrote:
>>>
>>>> Hi, team!
>>>>
>>>> Please look at the commit .
>>>>
>>>> Module 'mistral/model' now is responsible for object model
>>>> representation which is used for accessing properties of actions, tasks etc.
>>>>
>>>> We have a name problem - looks like we should rename module
>>>> 'mistral/model' since we have DB models and they are absolutely different.
>>>>
>>>>
>>>> Thoughts?
>>>>
>>>>
>>>> Best Regards,
>>>> Nikolay
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140305/ee383643/attachment.html>