Open Stack

On 22.12.2015 21:12, Yuriy Taraday wrote:
> Hello, everybody.
> 
> It's a week old thread and I should've jumped in earlier. Better late
> than never.
> 
> On Wed, Dec 16, 2015 at 2:04 AM Dmitriy Shulyak <dshulyak at mirantis.com
> <mailto:dshulyak at mirantis.com>> wrote:
> 
>     Hello folks,
> 
>     This topic is about configuration storage which will connect data
>     sources (nailgun/bareon/others) and orchestration. And right now we
>     are developing two projects that will overlap a bit.
> 
>     I understand there is not enough context to dive into this thread
>     right away, but i will appreciate if those people, who participated
>     in design, will add their opinions/clarifications on this matter.
> 
> 
> Let's try to add more context here. I see a lot of confusion around this
> matter. I think most of it comes from not having a single complete
> source of data about both approaches. I'll try to summarize the problem
> and outline proposed solutions as well as state of implementation for those.
> 
> == The problems. ==
> 
> Currently we have 2 main problems in question:
> 1. How to store data in Fuel so that it can come from different sources
> and be consumed by different pieces of software.
> 2. How to integrate Solar with Fuel and allow it to consume data
> provided by Nailgun (currently) or whatever else (if we get #1 implemented).

IIUC, that exposes Config Service to Solar as a "proxy" data source
among with other external sources like Nailgun, LDAP, IPAM and so on.
But there is support of external data sources on the Solar radar as
well. Developing "data resources" with "data processors" in Solar in
parallel with the Config Service doing literally the same -but as a
proxy data access layer- would bring major duplication of efforts, which
is not good. Instead, we should re-think of a "flat model" with no data
proxies and using Solar framework from the very beginning, IMHO.

I suggest to address this issue ASAP.

> 
> I was assigned (driven, actually) to look at the problem #1, and so with
> a start of a number of ideas from Oleg and others from my team and after
> some discussion with other people involved in Fuel architecture, I've
> finalized the scope and outlined architecture of the service that we
> would need to solve it in [0]. I didn't mean to step on anyone's toes,
> but later I was shown that the similar issue is being solved by SolarDB
> initiative that is being developed in the scope of integration between
> Solar and Fuel.
> 
> == The solutions. ==
> 
> = Config Service =
> 
> (because ConfigDB became an overused and ill-defined term)
> 
> Config Service is thought as a way to link Nailgun (and other sources in
> the future) to deployment tasks (and other consumers in the future) in a
> consistent and verifiable way. Someone lets the service know about the
> structure of the data that components provide and the structure of the
> data that other components consume thus declaring internal data flow
> between sources and consumers of data. Then for every environment
> Nailgun (for now - it can be other service, it can be changed to pull
> model later) feeds all necessary data into the service, triggers
> deployment tasks that consume data from the service the way that is more
> suitable for them. If we need to feed this data into some external
> service (Puppet master, for example), we're free to do so as long as we
> define data structure that a consumer expects.
> 
> = SolarDB =
> 
> (mainly based on [1] and presentations seen earlier, please correct me
> if smth wrong)
> 
> SolarDB includes active component: Data Processors (DPs). DPs fetch data
> from wherever they're intended to (Nailgun for starters, any other
> source in the future) and store them as Solar's Data Resorces (DRs). DRs
> are then translated to concrete data for other Solar's Resources
> (Executable Resources, ERs), this data is preprocessed by Policy Engine
> and converted to a set of calls to mutators that change ERs data in
> Solar's internal database in a way that lets Solar decide what should be
> done to change actual state of the environment.
> 
> == State of implementations ==
> 
> = Config Service =
> 
> I plan to show a PoC for Config Service integration before the end of
> this year. Coding of the service itself is almost at finish line at [2],
> integration with Nailgun and Astute/Puppet will take most of the
> remaining time.
> 
> = SolarDB =
> 
> PoC with Data Processors and Data Resources happened with simple cluster
> architecture (I don't know the date here). Policy Engine is in early
> stages of development.
> 
> == Main differences ==
> 
> I'll try to list main differences along with what seems for me pros and
> cons for both sides (major points taken from original Dmitriy's email).
> 
> 1. Config Service (CS) is initially planned as a passive element of
> integration while SolarDB (SD) have DPs that actively fetch data from
> the sources.
> 
> CS+: simpler implementation, can get PoC done fast
> CS+: easier to integrate with current Fuel architecture (actors remain
> actors there)
> CS+: can be easily modified to add active components to the service
> (pull model)
> CS-: brings in another component into the stack
> CS-: requires other components to be changed to push data into the service
> SD+: doesn't require any changes in other components
> SD-: requires Solar to store fetched data
> SD-: brings in another component into the stack
> 
> Overall I think that this point is not crucial here: both approaches can
> be converted back and forth without much effort in both solutions.
> 
> 2. Config Service is designed as an independent service, SolarDB is
> tight to Solar infrastructure.
> 
> CS+: defines clear interface between components
> CS+: doesn't require Solar at all, so we can land and use it independently
> CS-: duplicates data between Config Service and Solar
> SD+: integrated with Solar, reuses much of its infrastructure and avoids
> duplication of funcitonality
> SD+: has a clear and upfront way to feed necessary data to Solar
> (through Policy Engine)
> SD-: it's unclear how to use SolarDB when Solar is not in the picture,
> e.g. with external config management system.
> 
> 3. Config Service allows any consumer to provide a template that allows
> one to consume data in a way that's more suitable for it.
> 
> CS+: any external tool or system can consume data without need to
> convert it afterwards
> SD-: additional functionality would be required to integrate with
> external systems
> 
> 4. When it comes to overriding values or making changes in the schemas:
> 
> CS+: Config Service defines a REST API that user can use and build tools
> upon
> SD-: SolarDB seems to rely on editing YAML files stored somewhere to
> change data flow.
> 
> Clearly the biggest difference here is #2. Other items can be solved
> rather straightforward from both ends, so I'd like to avoid bikeshedding
> them here.
> 
>     Similar functionality
>     --------------------------
>     1. Hierachical storage
>     2. Versioning of changes
>     3. Possibility to overwrite config values
>     4. Schema for inputs
> 
>     Overall it seems that we share same goals for both services,
>     the difference lies in organizational and technical implementation
>     details.
> 
> 
> As outlined above, the main difference is that Config Service tries to
> solve one problem (#1) with no dependencies while SolarDB tries to solve
> both problems (#1 and #2) heavily depending on Solar.
> 
>     Possible solutions
>     ------------------------
>     1. develop configdb and solar with duplicated functionality
>     - at least 2 additional components will be added to the picture,
>     one is configdb, another one will need to sync data between configdb
>     and solar
>     - in some cases data in solar and configdb will be 100% duplicated
>     - different teams will work on same functionality
>     - integration of additional component for fuel will require
>     integration with
>     configdb and with solar
>     + configdb will be independent from solar orchestration/other components
> 
> 
> This seems to be the most viable way now, if we have clear integration
> plan for later.
> 
> From what I see, Config Service is to implement logic of SolarDB down to
> EPs data. So we can later use Config Service to generate EPs data that
> will be fed to Policy Engine, for example.
> 
> So during development and early integration, both projects can be used
> to solve different problems. After that all what's left is to integrate
> them with each other.
> 
>     2. make service out of solardb, allign with configdb use cases
>     + solardb will be independent from solar orchestration/other solar
>     components
>     + integration of fuel component will be easier than in 1st version
>     + clarity about components responsibility and new architecture
>     - redesign/refactoring communication between components in solar
> 
> 
> For me it seems that separating SolarDB from Solar will lead to the same
> effort that is currently being put into Config Service.
> 
>     3. do not use configdb/no extraction of solardb
>     - inproc communication, which can lead to coupled components (not
>     the case currently)
>     + faster implementation (no major changes required for integration
>     with fuel)
>     + clarity about components responsibility and new architecture
> 
> 
> The main con here is that we cannot land or use Config part without
> Solar. This eliminates possibility to give something to people to play
> and build tools on early. This will lead to strong coupling of Fuel with
> Solar which in the long run we want to avoid.
> 
>     Summary
>     -------------
>     For solar it makes no difference where data will come from: configdb or
>     data sources, but in overall fuel architecture it will lead to
>     significant
>     complexity increase.
>     It would be the best to follow 2nd path, because in long term we
>     don't want tightly coupled components, but in nearest future we need
>     to concentrate
>     on:
>     - integration with fuel
>     - implementing policy engine
>     - polishing solar components
>     This is why i am not sure that we can spend time on 2nd path right now,
>     or even before 9.0.
> 
> 
> For me, effort duplicated in solution #1 won't be a wasted time. Since
> DPs (seem to) work, from what Dmitriy outlined, only Policy Engine and
> Solar polishing are left for the effort to be put in. And for the time
> being we can have 2 parallel services because integrating them in the
> future shouldn't be a problem.
> 
> [0]
> https://docs.google.com/document/d/1I7nMOnSrmFhtMPhSWb8BJ48qtImmfUFemzNaNUmDvrY/edit?usp=sharing -
> Configuration Service Architecture (still a draft, not a final spec)
> [1] https://etherpad.openstack.org/p/solar-minisummit-fuel-integration-architecture
> [2] https://github.com/Mirantis/tuning-box - Config Service PoC code (to be)
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando