Open Stack

Hello, everybody.

It's a week old thread and I should've jumped in earlier. Better late than
never.

On Wed, Dec 16, 2015 at 2:04 AM Dmitriy Shulyak <dshulyak at mirantis.com>
wrote:

> Hello folks,
>
> This topic is about configuration storage which will connect data sources
> (nailgun/bareon/others) and orchestration. And right now we are developing
> two projects that will overlap a bit.
>
> I understand there is not enough context to dive into this thread right
> away, but i will appreciate if those people, who participated in design,
> will add their opinions/clarifications on this matter.
>

Let's try to add more context here. I see a lot of confusion around this
matter. I think most of it comes from not having a single complete source
of data about both approaches. I'll try to summarize the problem and
outline proposed solutions as well as state of implementation for those.

== The problems. ==

Currently we have 2 main problems in question:
1. How to store data in Fuel so that it can come from different sources and
be consumed by different pieces of software.
2. How to integrate Solar with Fuel and allow it to consume data provided
by Nailgun (currently) or whatever else (if we get #1 implemented).

I was assigned (driven, actually) to look at the problem #1, and so with a
start of a number of ideas from Oleg and others from my team and after some
discussion with other people involved in Fuel architecture, I've finalized
the scope and outlined architecture of the service that we would need to
solve it in [0]. I didn't mean to step on anyone's toes, but later I was
shown that the similar issue is being solved by SolarDB initiative that is
being developed in the scope of integration between Solar and Fuel.

== The solutions. ==

= Config Service =

(because ConfigDB became an overused and ill-defined term)

Config Service is thought as a way to link Nailgun (and other sources in
the future) to deployment tasks (and other consumers in the future) in a
consistent and verifiable way. Someone lets the service know about the
structure of the data that components provide and the structure of the data
that other components consume thus declaring internal data flow between
sources and consumers of data. Then for every environment Nailgun (for now
- it can be other service, it can be changed to pull model later) feeds all
necessary data into the service, triggers deployment tasks that consume
data from the service the way that is more suitable for them. If we need to
feed this data into some external service (Puppet master, for example),
we're free to do so as long as we define data structure that a consumer
expects.

= SolarDB =

(mainly based on [1] and presentations seen earlier, please correct me if
smth wrong)

SolarDB includes active component: Data Processors (DPs). DPs fetch data
from wherever they're intended to (Nailgun for starters, any other source
in the future) and store them as Solar's Data Resorces (DRs). DRs are then
translated to concrete data for other Solar's Resources (Executable
Resources, ERs), this data is preprocessed by Policy Engine and converted
to a set of calls to mutators that change ERs data in Solar's internal
database in a way that lets Solar decide what should be done to change
actual state of the environment.

== State of implementations ==

= Config Service =

I plan to show a PoC for Config Service integration before the end of this
year. Coding of the service itself is almost at finish line at [2],
integration with Nailgun and Astute/Puppet will take most of the remaining
time.

= SolarDB =

PoC with Data Processors and Data Resources happened with simple cluster
architecture (I don't know the date here). Policy Engine is in early stages
of development.

== Main differences ==

I'll try to list main differences along with what seems for me pros and
cons for both sides (major points taken from original Dmitriy's email).

1. Config Service (CS) is initially planned as a passive element of
integration while SolarDB (SD) have DPs that actively fetch data from the
sources.

CS+: simpler implementation, can get PoC done fast
CS+: easier to integrate with current Fuel architecture (actors remain
actors there)
CS+: can be easily modified to add active components to the service (pull
model)
CS-: brings in another component into the stack
CS-: requires other components to be changed to push data into the service
SD+: doesn't require any changes in other components
SD-: requires Solar to store fetched data
SD-: brings in another component into the stack

Overall I think that this point is not crucial here: both approaches can be
converted back and forth without much effort in both solutions.

2. Config Service is designed as an independent service, SolarDB is tight
to Solar infrastructure.

CS+: defines clear interface between components
CS+: doesn't require Solar at all, so we can land and use it independently
CS-: duplicates data between Config Service and Solar
SD+: integrated with Solar, reuses much of its infrastructure and avoids
duplication of funcitonality
SD+: has a clear and upfront way to feed necessary data to Solar (through
Policy Engine)
SD-: it's unclear how to use SolarDB when Solar is not in the picture, e.g.
with external config management system.

3. Config Service allows any consumer to provide a template that allows one
to consume data in a way that's more suitable for it.

CS+: any external tool or system can consume data without need to convert
it afterwards
SD-: additional functionality would be required to integrate with external
systems

4. When it comes to overriding values or making changes in the schemas:

CS+: Config Service defines a REST API that user can use and build tools
upon
SD-: SolarDB seems to rely on editing YAML files stored somewhere to change
data flow.

Clearly the biggest difference here is #2. Other items can be solved rather
straightforward from both ends, so I'd like to avoid bikeshedding them here.

Similar functionality
> --------------------------
> 1. Hierachical storage
> 2. Versioning of changes
> 3. Possibility to overwrite config values
> 4. Schema for inputs
>
> Overall it seems that we share same goals for both services,
> the difference lies in organizational and technical implementation details.
>

As outlined above, the main difference is that Config Service tries to
solve one problem (#1) with no dependencies while SolarDB tries to solve
both problems (#1 and #2) heavily depending on Solar.

Possible solutions
> ------------------------
> 1. develop configdb and solar with duplicated functionality
> - at least 2 additional components will be added to the picture,
> one is configdb, another one will need to sync data between configdb and
> solar
> - in some cases data in solar and configdb will be 100% duplicated
> - different teams will work on same functionality
> - integration of additional component for fuel will require integration
> with
> configdb and with solar
> + configdb will be independent from solar orchestration/other components
>

This seems to be the most viable way now, if we have clear integration plan
for later.

>From what I see, Config Service is to implement logic of SolarDB down to
EPs data. So we can later use Config Service to generate EPs data that will
be fed to Policy Engine, for example.

So during development and early integration, both projects can be used to
solve different problems. After that all what's left is to integrate them
with each other.

2. make service out of solardb, allign with configdb use cases
> + solardb will be independent from solar orchestration/other solar
> components
> + integration of fuel component will be easier than in 1st version
> + clarity about components responsibility and new architecture
> - redesign/refactoring communication between components in solar
>

For me it seems that separating SolarDB from Solar will lead to the same
effort that is currently being put into Config Service.

3. do not use configdb/no extraction of solardb
> - inproc communication, which can lead to coupled components (not the case
> currently)
> + faster implementation (no major changes required for integration with
> fuel)
> + clarity about components responsibility and new architecture
>

The main con here is that we cannot land or use Config part without Solar.
This eliminates possibility to give something to people to play and build
tools on early. This will lead to strong coupling of Fuel with Solar which
in the long run we want to avoid.

Summary
> -------------
> For solar it makes no difference where data will come from: configdb or
> data sources, but in overall fuel architecture it will lead to significant
> complexity increase.
> It would be the best to follow 2nd path, because in long term we don't
> want tightly coupled components, but in nearest future we need to
> concentrate
> on:
> - integration with fuel
> - implementing policy engine
> - polishing solar components
> This is why i am not sure that we can spend time on 2nd path right now,
> or even before 9.0.
>

For me, effort duplicated in solution #1 won't be a wasted time. Since DPs
(seem to) work, from what Dmitriy outlined, only Policy Engine and Solar
polishing are left for the effort to be put in. And for the time being we
can have 2 parallel services because integrating them in the future
shouldn't be a problem.

[0]
https://docs.google.com/document/d/1I7nMOnSrmFhtMPhSWb8BJ48qtImmfUFemzNaNUmDvrY/edit?usp=sharing
-
Configuration Service Architecture (still a draft, not a final spec)
[1]
https://etherpad.openstack.org/p/solar-minisummit-fuel-integration-architecture
[2] https://github.com/Mirantis/tuning-box - Config Service PoC code (to be)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151222/8a5ece1c/attachment.html>