Open Stack

I definitely agree with what Evgeniy said.

@Oleg could you make a step-by-step how do you imagine this
integration (with separate ConfigDB) ? To me this adds at least one
component / integration point.


> The point is that for Solar integration, we still need integration points, and the less of them we have, the more simple the transition is going to be..

With DR+DP you will have exactly one integration point.

> I also don't like polling model for data processors. In my understanding, components should push their changes through the pipeline. Although this is pure implementation detail and is not really important ATM.

Push is problematic because:
1) if you don't own all components then you're in troubles
2) you never know how valid are data of component (you could add some
time based markers, but still it's not enough), you basically loose
control about data. With pull model you know more about context etc.

On Mon, Dec 21, 2015 at 11:49 AM, Evgeniy L <eli at mirantis.com> wrote:
> Comments inline.
>
> On Mon, Dec 21, 2015 at 1:42 PM, Oleg Gelbukh <ogelbukh at mirantis.com> wrote:
>>
>> The problem with this approach is that we can't manage the interfaces
>> between components without changing the code of 2+ components (i.e. one that
>> provides the data and all that consume it).
>
>
> I'm not sure if understand you correctly, none of these approaches
> require to change the components, we will have change a single
> place, which is specific data processor.
>
>>
>>
>> I also don't like polling model for data processors. In my understanding,
>> components should push their changes through the pipeline. Although this is
>> pure implementation detail and is not really important ATM.
>
>
> We don't have any other choice, we don't control components, configuration
> components are not going to implement Fuel specific logic, so Fuel has
> to pool the data.
>
>>
>>
>> The point is that for Solar integration, we still need integration points,
>> and the less of them we have, the more simple the transition is going to
>> be..
>
>
> As described above, there will be a single integration point, data
> processor.
>>
>>
>> --
>> Best regards,
>> Oleg Gelbukh
>>
>> On Mon, Dec 21, 2015 at 11:32 AM, Evgeniy L <eli at mirantis.com> wrote:
>>>
>>> Hi Oleg,
>>>
>>> I understand the concern, but in case of integration specifically with
>>> Solar,
>>> I don't see any reasons to add ConfigDB, because Solar by itself is a
>>> ConfigDB.
>>> At the same time I would agree that there might be a case, when user uses
>>> Zookeeper/Puppet Master/CMDB as a data store, in this case we should
>>> store
>>> the data directly in those services, without keeping them in yet another
>>> storage.
>>>
>>> So the flow will look like this:
>>> Components ->
>>> Data get polled by data processors ->
>>> | Solar data processor puts the data into Solar in its format
>>> | Zookeeper data processor puts the data into Zookeeper in its format
>>> | Custom CMDB data processor puts the data into CMDB in its own format
>>>
>>> Thanks,
>>>
>>> On Fri, Dec 18, 2015 at 7:00 PM, Oleg Gelbukh <ogelbukh at mirantis.com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> The idea behind configdb is that it is independent component that
>>>> defines data flows between other components of the system. It has no
>>>> knowledge about those components or specifics of their data. Data formats
>>>> are defined by components themselves via schemas/templates and can be
>>>> changed at any time (i.e. don't require code changes).
>>>>
>>>> Important 'pro' having ConfigDB separate from Solar is that it will
>>>> simplify transition from current Fuel architecture by breaking it into more
>>>> definite stages and reducing the number of components Solar have to be
>>>> integrated with.
>>>>
>>>> --
>>>> Best regards,
>>>> Oleg Gelbukh
>>>>
>>>> On Wed, Dec 16, 2015 at 4:38 PM, Evgeniy L <eli at mirantis.com> wrote:
>>>>>
>>>>> Hi Dmitry,
>>>>>
>>>>> I also don't think that we should duplicate the data in configdb,
>>>>> because in this case there will be +2 additional interfaces which
>>>>> will require to covert the data into configdb and after that from
>>>>> configdb to Solar, which seems redundant overhead.
>>>>>
>>>>> But we should be able to put the data directly to user's
>>>>> CMDB/ZooKeeper/Puppet Master/etc.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> On Wed, Dec 16, 2015 at 2:03 AM, Dmitriy Shulyak
>>>>> <dshulyak at mirantis.com> wrote:
>>>>>>
>>>>>> Hello folks,
>>>>>>
>>>>>> This topic is about configuration storage which will connect data
>>>>>> sources (nailgun/bareon/others) and orchestration. And right now we are
>>>>>> developing two projects that will overlap a bit.
>>>>>>
>>>>>> I understand there is not enough context to dive into this thread
>>>>>> right away, but i will appreciate if those people, who participated in
>>>>>> design, will add their opinions/clarifications on this matter.
>>>>>>
>>>>>> Main disagreements
>>>>>> ---------------------------
>>>>>> 1. configdb should be passive, writing to configdb is someone else
>>>>>> responsibility
>>>>>> + simpler implementation, easier to use
>>>>>> - we will need another component that will do writing, or split this
>>>>>> responsibility somehow
>>>>>>
>>>>>> 2. can be used without other solar components
>>>>>> + clear inteface between solar components and storage layer
>>>>>> - additional work required to design/refactor communication layer
>>>>>> between modules in solar
>>>>>> - some data will be duplicated between solar orchestrator layer and
>>>>>> configdb
>>>>>>
>>>>>> 3. templates for output
>>>>>> technical detail, can be added on top of solardb if required
>>>>>>
>>>>>> Similar functionality
>>>>>> --------------------------
>>>>>> 1. Hierachical storage
>>>>>> 2. Versioning of changes
>>>>>> 3. Possibility to overwrite config values
>>>>>> 4. Schema for inputs
>>>>>>
>>>>>> Overall it seems that we share same goals for both services,
>>>>>> the difference lies in organizational and technical implementation
>>>>>> details.
>>>>>>
>>>>>> Possible solutions
>>>>>> ------------------------
>>>>>> 1. develop configdb and solar with duplicated functionality
>>>>>> - at least 2 additional components will be added to the picture,
>>>>>> one is configdb, another one will need to sync data between configdb
>>>>>> and solar
>>>>>> - in some cases data in solar and configdb will be 100% duplicated
>>>>>> - different teams will work on same functionality
>>>>>> - integration of additional component for fuel will require
>>>>>> integration with
>>>>>> configdb and with solar
>>>>>> + configdb will be independent from solar orchestration/other
>>>>>> components
>>>>>>
>>>>>> 2. make service out of solardb, allign with configdb use cases
>>>>>> + solardb will be independent from solar orchestration/other solar
>>>>>> components
>>>>>> + integration of fuel component will be easier than in 1st version
>>>>>> + clarity about components responsibility and new architecture
>>>>>> - redesign/refactoring communication between components in solar
>>>>>>
>>>>>> 3. do not use configdb/no extraction of solardb
>>>>>> - inproc communication, which can lead to coupled components (not the
>>>>>> case currently)
>>>>>> + faster implementation (no major changes required for integration
>>>>>> with fuel)
>>>>>> + clarity about components responsibility and new architecture
>>>>>>
>>>>>> Summary
>>>>>> -------------
>>>>>> For solar it makes no difference where data will come from: configdb
>>>>>> or
>>>>>> data sources, but in overall fuel architecture it will lead to
>>>>>> significant
>>>>>> complexity increase.
>>>>>> It would be the best to follow 2nd path, because in long term we don't
>>>>>> want tightly coupled components, but in nearest future we need to
>>>>>> concentrate
>>>>>> on:
>>>>>> - integration with fuel
>>>>>> - implementing policy engine
>>>>>> - polishing solar components
>>>>>> This is why i am not sure that we can spend time on 2nd path right
>>>>>> now,
>>>>>> or even before 9.0.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> __________________________________________________________________________
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
--
Warm regards,
Jędrzej Nowak