Open Stack

Thanks Vladimir.
This is very important work, I'd say. I'd split it into two parts:

   1. Have some store where you'd dump data from serializers. Hiera should
   be able to read easily directly from this store.
   2. Refactor serializers so to get rid of single python method which
   creates data for multiple OpenStack components, and allow deployment
   engineers to easily modify code of particular piece

For #1, it is important to think broadly. We want this store to be used by
other tools which users may have (Puppet Master, Ansible, etc.) as a source
data, and so that Fuel & other tools can coexist on the same env if really
needed (even though I'd ideally try to avoid it).
We need to have an abstraction layer there, so that we can have drivers for
key-value store and for such things as Zookeeper, for instance, in the
future. I think we need to address #1 in the first order, before going to
#2 (if we can't do it in parallel).

For #2, I think we need to consider flexibility. What if we use Ansible, or
container for some of our services? So we need to think where we can put
these per-component / per-task serializers, so those can be used for both
Puppet module & something different.

Also, it's interesting problem from the execution point of view. Do we run
serialization on Fuel Master side or on slave nodes, where we install
OpenStack? I see some issues with running it on OpenStack nodes, even
though I like an idea of load distribution, etc. For instance, if you run
almost all graph, and then the last task in the graph runs corresponding
serializer - and there is a Python exception for whatever reason (user
input leads to bug in calculation). You could get it right a way, if you
tried to calculate it before overall deployment - but now you've been
waiting deployment to be almost done to catch it.

Thank you,

On Fri, Oct 16, 2015 at 9:22 AM Vladimir Kuklin <vkuklin at mirantis.com>
wrote:

> Hey, Fuelers
>
> TL;DR This email is about how to make
>
> * Intro
> I want to bring up one of the important topics on how to make Fuel more
> flexible. Some of you know that we have been discussing means of doing this
> internally and now it is time to share these thoughts with all of you.
>
> As you could know per Evgeniy Li's message [0] we are looking forward
> splitting Fuel (specifically it's Fuel-Web) part into set of microservices
> each one serving their own purpose like networking configuration,
> partitioning, etc.
>
>
> And while we are working on this it seems that we need to get rid of
> so-called Nailgun serializers that are put too close to business logic
> engine, that have a lot of duplicating attributes; you are not able to
> easily modify or extend them; you are not able to change their behaviour
> even when Fuel Library is capable of doing so - everything is hardcoded in
> Nailgun code without clear separation between business logic and actual
> deployment workflow data generation and orchestration.
>
> Let me give you an example:
>
> * Case A. Replace Linux bridges with OVS bridges by default
>
> We all know that we removed OVS as much as possible from our reference
> architecture due to its buginess. Imagine a situation when someone
> magically fixed OVS and wants to use it as a provider for generic bonds and
> bridge. It actually means that he needs to set default provider in
> network_scheme for l23network puppet module to 'ovs' instead of 'lnx'.
> Imagine, he has put this magical OVS into a package and created a plugin.
> The problem here will be that he needs to override what network serializer
> is sending to the nodes.
>
> But the problem here is that he cannot do it without editing Nailgun code
> or override this serializer in any way.
>
> * Case B. Make Swift Partitions Known to Fuel Library
>
> Imagine, you altered the way you partition your disk in Nailgun. You
> created a special role for swift disks which should occupy the whole disk.
> In this case you should be able to get this info from api and feed it to
> swift deployment task. But it is not so easy - this stuff is still
> hardcoded in deployment serializers like {mp} field of nodes array of
> hashes.
>
> * Proposed solution
>
> In order to tackle this I propose to extract these so called serializers
> (see links [1] and [2]) and put them closer to library. You can see that
> half of the code is actually duplicated for deployment and provsioning
> serializers and there is actually no inheritance of common code betwen
> them. If you want to introduce new attribute and put it into astute.yaml,
> you will need to rewrite Nailgun code. This is not very
> deployment/sysop/sysadmin engineer-friendly. Essentially, the proposal is
> to introduce a library of such `serializers` (I would like to call them
> translators actually) which could leverage inheritance, polymorphism and
> incapsulation pretty much in OOP mode but with ability for deployment
> engineers to apply versioning to serializers and allow each particular task
> to work with different sources of data with different versions of API.
>
> What this actually means: each task has a step called 'translation' which
> fetches attributes from any arbitrary set of sources and converts them into
> the format that is consumable by the deployment stage of this task. From
> our current architectural point of view it will look like generation of a
> set of yaml files that will be merged by hiera so that each puppet task can
> leverage the power of hiera.
>
> This actually means that in scope of our modularization initiative each
> module should have an API which will be accessed by those tasks in runtime
> right before the tasks are executed. This also means that if a user changes
> some of the values in the databases of those modules, rerun of such task
> will lead to a different result of 'translation' and trigger some actions
> like 'keystone_config ~> Service[keystone]' in puppet.
>
> There is a tough discussion (etherpad here:[4]) on:
>
> 1) how to handle versioning/revert capabilities
> 2) where to store output produced by those 'translators'
> 3) which type of the storage to use
>
> Please, feel free to provide your feedback on this approach and tell me
> where this approach is going to be wrong.
>
> [0] http://permalink.gmane.org/gmane.comp.cloud.openstack.devel/66563
> [1]
> https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/orchestrator/deployment_serializers.py
> [2]
> https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/orchestrator/provisioning_serializers.py
> [3] https://github.com/xenolog/l23network
> [4] https://etherpad.openstack.org/p/data-processor-per-component
>
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com <http://www.mirantis.ru/>
> www.mirantis.ru
> vkuklin at mirantis.com
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 
Mike Scherbakov
#mihgen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151020/f721d50a/attachment.html>