[openstack-dev] [Fuel][Fuel-Modularization] Proposal on Decoupling Serializers from Nailgun

Oleg Gelbukh ogelbukh at mirantis.com
Thu Oct 22 10:20:03 UTC 2015


Hi Vladimir,

Thanks for prompt reply. Please, see my comments inline.

On Thu, Oct 22, 2015 at 12:44 PM, Vladimir Kuklin <vkuklin at mirantis.com>
wrote:

> Oleg
>
> Thank you for your feedback. IMO, the schema you are providing is very
> complex and would surely benefit from some examples.
>

I'm going to submit spec for review that will incorporate examples and
diagrams for sure. I expect to come up with it in couple of days, most
likely by Monday.

>
> If I understand correctly your proposal, you are trying to do the things
> that we actually want to get rid of - tight coupling and schema control of
> data that is being used by components.
>

Your understanding is mostly correct. However, important thing here is that
we propose API that will allow to adjust schema of the particular view at
any time or register a new schema (for new/added component), etc, (almost)
without writing Python code.


> There should be no cross-reference between components that do actual
> deployment. Instead, there should be a clear separation between layers of
> our deployment system.
>

Such a separation will not be enforced by the system we propose. However,
if we indeed have some 'hierarchy' of components, it will be naturally
reflected in the way the links are specified in templates. For example, if
our primary source of configuration settings is UI/API, then it will be
authoritative for configurable parameters, like backends selection, IP
address ranges, etc. However, settings that are discovered from actual
nodes shall be provided by corresponding components, like 'nailgun-agent'.

Deployment modules most likely won't be authoritative for any settings, as
far as I can tell at the moment. They could, however, provide feedback-like
parameters, for instance, those that can be calculated only in runtime.


>
> All the data that is provided to deployment (or
> provisioning/power_management/etc.) tasks should be accessible through API
> of the top-level components such as
> Network/Partitioning/IPAddressAllocation/ Manager or any other type of
> external configuration database such as ENC of external puppet
> master/LDAP/<any other inventory system>.
>

This very proposal is about creating such an API (whether service-like or
library-like) that other components and even end users can leverage to
access and manage configuration parameters. We probably should start with
library API, and decide whether we need service of this kind later.


>
>  Each task can use some code to transform this output to the
> representation that is actually needed for this particular task. Whenever a
> task transforms this data it can access API and do version negotiation, for
> example. Each time this transformation is performed this task can return
> the data to some storage that will save this data for sake of control and
> troubleshooting, such as, for example, user can always see which changes
> are going to be applied and decide what to do next.
>
> Also, this means that the process of data calculation itself is very
> 'lazy' or 'delayed', i. e. the data itself is calculated right at the
> beginning of deployment transaction, so that it is not locked to some
> particular details of deployment engine data processing and not prone to
> issues like 'oh, I cannot get VIP because it has not been allocated yet by
> Nailgun/oh, I cannot set it because it has already been set by Nailgun and
> there is no way to alter it'.
>

To me, the two paragraphs above a contradictory. If the data calculations
are lazy, I don't really see how one can introspect into changes that will
be applied by a component at any given run. You just don't have this
information, and you need to calculate it anyways to see which settings
will be passed to a component. Might be I got your point wrong here. Please
correct me if this is the case.

Thanks again, looking forward to hear from you.

--
Best regards,
Oleg Gelbukh


>
> On Thu, Oct 22, 2015 at 12:16 PM, Oleg Gelbukh <ogelbukh at mirantis.com>
> wrote:
>
>> Hello,
>>
>> We discussed this proposal in our team and came up with the following
>> vision of a configuration provisioning system:
>>
>>     - Installer is a system of multiple components: hardware inventory,
>> user interfaces, provisioning modules, deployment modules, checker modules,
>> volume manager, network manager, plugins, etc.
>>     - Every component has its own data representation (we call them
>> 'views' as they provide an introspection in the configuration of the
>> system), which should include all the settings data the component should
>> have access to to perform its functions.
>>     - Every component has 2 types of data in its view/representation:
>> authoritative data (which the component can modify) and external data
>> (which essentially is links to elements of another component's
>> view/representation).
>>     - There is no 'universal' or 'general' representation of data which
>> serves a source of truth for all other views: every component is a source
>> of truth for its authoritative data.
>>     - Views are defined as templates in some declarative language (YaML,
>> JSON, XML, %whatever%), think of jsonschema here. Authoritative settings of
>> the component have only type, external settings must also contain a link to
>> external view (might be just piece of code with properly referenced
>> elements of external view as parameters).
>>     - View template shall be rendered in the data store during
>> 'registration' of the component in the system, i.e. data structure shall be
>> created to represent the format of the data with necessary links.
>>     - Views can be saved to the data store and modified by component that
>> 'owns' the view's template, or via system's API. Changes to authoritative
>> settings in the view shall be propagated to all views that contain external
>> links to those settings.
>>     - Both view template and views defined by it have versions. Template
>> version if defined by the version of it's owner component. View version
>> increases with every change made to it and can be used by the orchestrator
>> and component to determine if the async update of view was made by external
>> links.
>>
>> We will continue to flesh it out as a specification in Fuel specs
>> repository. I will greatly appreciate any feedback on this vision,
>> including comments, objections, concerns and questions.
>>
>> --
>> Best regards,
>> Oleg Gelbukh
>>
>> On Tue, Oct 20, 2015 at 2:13 PM, Vladimir Kuklin <vkuklin at mirantis.com>
>> wrote:
>>
>>> Folks
>>>
>>> Can we please stop using etherpad and move to some more usable thing as
>>> Google Docs? Etherpad seems too unusable for such discussion especially
>>> with this coloured formatting.
>>>
>>> Mike
>>>
>>> I currently see no need in following marketing trend for noSQL here - we
>>> need to store a set of structured data. This store should be the one that
>>> can be easily consumed directly or with some API wrapper. That is all. We
>>> will need to carefully evaluate each storage engine and decide which to
>>> pick. I personally insist on the engine that provides 100% consistency
>>> which is in fact opposite to what most of noSQL and distributed
>>> architectures provide. Nobody cares if you lose 1 billion of messages in a
>>> social network (even these messages authors) - this is almost all the time
>>> garbage with porn and cat pictures. Things will get worse if you destroy
>>> something in production serving accounting in your cloud due to the fact
>>> that nodes are
>>>
>>> I agree with option #2 - we actually should have task abstraction layer
>>> with drivers for execution, but I would go with baby steps for supporting
>>> other deployment tools - currently I do not see any benefit in using
>>> Ansible for tasks that Fuel is solving. The same is almost true for
>>> containers, but this is a different story.
>>>
>>> Eugene, Mike
>>>
>>> I agree with you that we need to think about where to execute these
>>> serializers. I think that we could do it the following way - serializer can
>>> be executed wherever it can actually work and it should possibly put data
>>> into centralized storage for the means of logging, control and accounting.
>>> I am not sure that this is the limitation case all the users will agree
>>> with, but we need to think of it.
>>>
>>> Regarding this 'last task throwing an exception issue' - we can handle
>>> this properly by simply rerunning the task that failed only due to
>>> serialization problem. Or even better - reorder its execution for later
>>> steps and try it again in a while if there are other tasks to be executed.
>>>
>>> But Mike's approach of data preparation prior to deployment/workflow
>>> transaction execution seems more viable. I think, we should follow the
>>> following one: "If you do not know the data before the transaction run,
>>> this data should be calculated after this transaction ends and this data
>>> should be used for another workflow in a different transaction".
>>>
>>>
>>> On Tue, Oct 20, 2015 at 1:20 PM, Evgeniy L <eli at mirantis.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a comment regarding to when/where run translators.
>>>> I think data processing (fetching + validation + translation) should be
>>>> done
>>>> as a separate stage, this way when we start deployment, we are sure
>>>> that we have everything we need to perform deployment, but I understand
>>>> that there might be exceptions and sometimes we will have to get the
>>>> data on the fly.
>>>> If we are going to put translated data into some distributed store I'm
>>>> not sure
>>>> if distributed approach for fetching and updating the data won't cause
>>>> the problems
>>>> with race conditions, in centralised approach probability to get such
>>>> problems can
>>>> be reduced.
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> On Tue, Oct 20, 2015 at 5:14 AM, Mike Scherbakov <
>>>> mscherbakov at mirantis.com> wrote:
>>>>
>>>>> Thanks Vladimir.
>>>>> This is very important work, I'd say. I'd split it into two parts:
>>>>>
>>>>>    1. Have some store where you'd dump data from serializers. Hiera
>>>>>    should be able to read easily directly from this store.
>>>>>    2. Refactor serializers so to get rid of single python method
>>>>>    which creates data for multiple OpenStack components, and allow deployment
>>>>>    engineers to easily modify code of particular piece
>>>>>
>>>>> For #1, it is important to think broadly. We want this store to be
>>>>> used by other tools which users may have (Puppet Master, Ansible, etc.) as
>>>>> a source data, and so that Fuel & other tools can coexist on the same env
>>>>> if really needed (even though I'd ideally try to avoid it).
>>>>> We need to have an abstraction layer there, so that we can have
>>>>> drivers for key-value store and for such things as Zookeeper, for instance,
>>>>> in the future. I think we need to address #1 in the first order, before
>>>>> going to #2 (if we can't do it in parallel).
>>>>>
>>>>> For #2, I think we need to consider flexibility. What if we use
>>>>> Ansible, or container for some of our services? So we need to think where
>>>>> we can put these per-component / per-task serializers, so those can be used
>>>>> for both Puppet module & something different.
>>>>>
>>>>> Also, it's interesting problem from the execution point of view. Do we
>>>>> run serialization on Fuel Master side or on slave nodes, where we install
>>>>> OpenStack? I see some issues with running it on OpenStack nodes, even
>>>>> though I like an idea of load distribution, etc. For instance, if you run
>>>>> almost all graph, and then the last task in the graph runs corresponding
>>>>> serializer - and there is a Python exception for whatever reason (user
>>>>> input leads to bug in calculation). You could get it right a way, if you
>>>>> tried to calculate it before overall deployment - but now you've been
>>>>> waiting deployment to be almost done to catch it.
>>>>>
>>>>> Thank you,
>>>>>
>>>>> On Fri, Oct 16, 2015 at 9:22 AM Vladimir Kuklin <vkuklin at mirantis.com>
>>>>> wrote:
>>>>>
>>>>>> Hey, Fuelers
>>>>>>
>>>>>> TL;DR This email is about how to make
>>>>>>
>>>>>> * Intro
>>>>>> I want to bring up one of the important topics on how to make Fuel
>>>>>> more flexible. Some of you know that we have been discussing means of doing
>>>>>> this internally and now it is time to share these thoughts with all of you.
>>>>>>
>>>>>> As you could know per Evgeniy Li's message [0] we are looking forward
>>>>>> splitting Fuel (specifically it's Fuel-Web) part into set of microservices
>>>>>> each one serving their own purpose like networking configuration,
>>>>>> partitioning, etc.
>>>>>>
>>>>>>
>>>>>> And while we are working on this it seems that we need to get rid of
>>>>>> so-called Nailgun serializers that are put too close to business logic
>>>>>> engine, that have a lot of duplicating attributes; you are not able to
>>>>>> easily modify or extend them; you are not able to change their behaviour
>>>>>> even when Fuel Library is capable of doing so - everything is hardcoded in
>>>>>> Nailgun code without clear separation between business logic and actual
>>>>>> deployment workflow data generation and orchestration.
>>>>>>
>>>>>> Let me give you an example:
>>>>>>
>>>>>> * Case A. Replace Linux bridges with OVS bridges by default
>>>>>>
>>>>>> We all know that we removed OVS as much as possible from our
>>>>>> reference architecture due to its buginess. Imagine a situation when
>>>>>> someone magically fixed OVS and wants to use it as a provider for generic
>>>>>> bonds and bridge. It actually means that he needs to set default provider
>>>>>> in network_scheme for l23network puppet module to 'ovs' instead of 'lnx'.
>>>>>> Imagine, he has put this magical OVS into a package and created a plugin.
>>>>>> The problem here will be that he needs to override what network serializer
>>>>>> is sending to the nodes.
>>>>>>
>>>>>> But the problem here is that he cannot do it without editing Nailgun
>>>>>> code or override this serializer in any way.
>>>>>>
>>>>>> * Case B. Make Swift Partitions Known to Fuel Library
>>>>>>
>>>>>> Imagine, you altered the way you partition your disk in Nailgun. You
>>>>>> created a special role for swift disks which should occupy the whole disk.
>>>>>> In this case you should be able to get this info from api and feed it to
>>>>>> swift deployment task. But it is not so easy - this stuff is still
>>>>>> hardcoded in deployment serializers like {mp} field of nodes array of
>>>>>> hashes.
>>>>>>
>>>>>> * Proposed solution
>>>>>>
>>>>>> In order to tackle this I propose to extract these so called
>>>>>> serializers (see links [1] and [2]) and put them closer to library. You can
>>>>>> see that half of the code is actually duplicated for deployment and
>>>>>> provsioning serializers and there is actually no inheritance of common code
>>>>>> betwen them. If you want to introduce new attribute and put it into
>>>>>> astute.yaml, you will need to rewrite Nailgun code. This is not very
>>>>>> deployment/sysop/sysadmin engineer-friendly. Essentially, the proposal is
>>>>>> to introduce a library of such `serializers` (I would like to call them
>>>>>> translators actually) which could leverage inheritance, polymorphism and
>>>>>> incapsulation pretty much in OOP mode but with ability for deployment
>>>>>> engineers to apply versioning to serializers and allow each particular task
>>>>>> to work with different sources of data with different versions of API.
>>>>>>
>>>>>> What this actually means: each task has a step called 'translation'
>>>>>> which fetches attributes from any arbitrary set of sources and converts
>>>>>> them into the format that is consumable by the deployment stage of this
>>>>>> task. From our current architectural point of view it will look like
>>>>>> generation of a set of yaml files that will be merged by hiera so that each
>>>>>> puppet task can leverage the power of hiera.
>>>>>>
>>>>>> This actually means that in scope of our modularization initiative
>>>>>> each module should have an API which will be accessed by those tasks in
>>>>>> runtime right before the tasks are executed. This also means that if a user
>>>>>> changes some of the values in the databases of those modules, rerun of such
>>>>>> task will lead to a different result of 'translation' and trigger some
>>>>>> actions like 'keystone_config ~> Service[keystone]' in puppet.
>>>>>>
>>>>>> There is a tough discussion (etherpad here:[4]) on:
>>>>>>
>>>>>> 1) how to handle versioning/revert capabilities
>>>>>> 2) where to store output produced by those 'translators'
>>>>>> 3) which type of the storage to use
>>>>>>
>>>>>> Please, feel free to provide your feedback on this approach and tell
>>>>>> me where this approach is going to be wrong.
>>>>>>
>>>>>> [0] http://permalink.gmane.org/gmane.comp.cloud.openstack.devel/66563
>>>>>> [1]
>>>>>> https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/orchestrator/deployment_serializers.py
>>>>>> [2]
>>>>>> https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/orchestrator/provisioning_serializers.py
>>>>>> [3] https://github.com/xenolog/l23network
>>>>>> [4] https://etherpad.openstack.org/p/data-processor-per-component
>>>>>>
>>>>>> --
>>>>>> Yours Faithfully,
>>>>>> Vladimir Kuklin,
>>>>>> Fuel Library Tech Lead,
>>>>>> Mirantis, Inc.
>>>>>> +7 (495) 640-49-04
>>>>>> +7 (926) 702-39-68
>>>>>> Skype kuklinvv
>>>>>> 35bk3, Vorontsovskaya Str.
>>>>>> Moscow, Russia,
>>>>>> www.mirantis.com <http://www.mirantis.ru/>
>>>>>> www.mirantis.ru
>>>>>> vkuklin at mirantis.com
>>>>>>
>>>>>> __________________________________________________________________________
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>> --
>>>>> Mike Scherbakov
>>>>> #mihgen
>>>>>
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>>
>>> --
>>> Yours Faithfully,
>>> Vladimir Kuklin,
>>> Fuel Library Tech Lead,
>>> Mirantis, Inc.
>>> +7 (495) 640-49-04
>>> +7 (926) 702-39-68
>>> Skype kuklinvv
>>> 35bk3, Vorontsovskaya Str.
>>> Moscow, Russia,
>>> www.mirantis.com <http://www.mirantis.ru/>
>>> www.mirantis.ru
>>> vkuklin at mirantis.com
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com <http://www.mirantis.ru/>
> www.mirantis.ru
> vkuklin at mirantis.com
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151022/bc446e37/attachment.html>


More information about the OpenStack-dev mailing list