<div dir="ltr">Hi Vladimir,<div><br></div><div>Thanks for prompt reply. Please, see my comments inline.<br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 22, 2015 at 12:44 PM, Vladimir Kuklin <span dir="ltr"><<a href="mailto:vkuklin@mirantis.com" target="_blank">vkuklin@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Oleg<div><br></div><div>Thank you for your feedback. IMO, the schema you are providing is very complex and would surely benefit from some examples.</div></div></blockquote><div><br></div><div>I'm going to submit spec for review that will incorporate examples and diagrams for sure. I expect to come up with it in couple of days, most likely by Monday.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>If I understand correctly your proposal, you are trying to do the things that we actually want to get rid of - tight coupling and schema control of data that is being used by components. </div></div></blockquote><div><br></div><div>Your understanding is mostly correct. However, important thing here is that we propose API that will allow to adjust schema of the particular view at any time or register a new schema (for new/added component), etc, (almost) without writing Python code.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>There should be no cross-reference between components that do actual deployment. Instead, there should be a clear separation between layers of our deployment system. </div></div></blockquote><div><br></div><div>Such a separation will not be enforced by the system we propose. However, if we indeed have some 'hierarchy' of components, it will be naturally reflected in the way the links are specified in templates. For example, if our primary source of configuration settings is UI/API, then it will be authoritative for configurable parameters, like backends selection, IP address ranges, etc. However, settings that are discovered from actual nodes shall be provided by corresponding components, like 'nailgun-agent'.</div><div><br></div><div>Deployment modules most likely won't be authoritative for any settings, as far as I can tell at the moment. They could, however, provide feedback-like parameters, for instance, those that can be calculated only in runtime.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>All the data that is provided to deployment (or provisioning/power_management/etc.) tasks should be accessible through API of the top-level components such as Network/Partitioning/IPAddressAllocation/ Manager or any other type of external configuration database such as ENC of external puppet master/LDAP/<any other inventory system>.</div></div></blockquote><div><br></div><div>This very proposal is about creating such an API (whether service-like or library-like) that other components and even end users can leverage to access and manage configuration parameters. We probably should start with library API, and decide whether we need service of this kind later.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div> Each task can use some code to transform this output to the representation that is actually needed for this particular task. Whenever a task transforms this data it can access API and do version negotiation, for example. Each time this transformation is performed this task can return the data to some storage that will save this data for sake of control and troubleshooting, such as, for example, user can always see which changes are going to be applied and decide what to do next. </div><div><br></div><div>Also, this means that the process of data calculation itself is very 'lazy' or 'delayed', i. e. the data itself is calculated right at the beginning of deployment transaction, so that it is not locked to some particular details of deployment engine data processing and not prone to issues like 'oh, I cannot get VIP because it has not been allocated yet by Nailgun/oh, I cannot set it because it has already been set by Nailgun and there is no way to alter it'.</div></div></blockquote><div><br></div><div>To me, the two paragraphs above a contradictory. If the data calculations are lazy, I don't really see how one can introspect into changes that will be applied by a component at any given run. You just don't have this information, and you need to calculate it anyways to see which settings will be passed to a component. Might be I got your point wrong here. Please correct me if this is the case.</div><div><br></div><div>Thanks again, looking forward to hear from you.</div><div><br></div><div>--</div><div>Best regards,</div><div>Oleg Gelbukh</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class=""><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 22, 2015 at 12:16 PM, Oleg Gelbukh <span dir="ltr"><<a href="mailto:ogelbukh@mirantis.com" target="_blank">ogelbukh@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>We discussed this proposal in our team and came up with the following vision of a configuration provisioning system:</div><div><br></div> - Installer is a system of multiple components: hardware inventory, user interfaces, provisioning modules, deployment modules, checker modules, volume manager, network manager, plugins, etc.<br> - Every component has its own data representation (we call them 'views' as they provide an introspection in the configuration of the system), which should include all the settings data the component should have access to to perform its functions.<br> - Every component has 2 types of data in its view/representation: authoritative data (which the component can modify) and external data (which essentially is links to elements of another component's view/representation).<br> - There is no 'universal' or 'general' representation of data which serves a source of truth for all other views: every component is a source of truth for its authoritative data.<br> - Views are defined as templates in some declarative language (YaML, JSON, XML, %whatever%), think of jsonschema here. Authoritative settings of the component have only type, external settings must also contain a link to external view (might be just piece of code with properly referenced elements of external view as parameters).<br> - View template shall be rendered in the data store during 'registration' of the component in the system, i.e. data structure shall be created to represent the format of the data with necessary links.<br> - Views can be saved to the data store and modified by component that 'owns' the view's template, or via system's API. Changes to authoritative settings in the view shall be propagated to all views that contain external links to those settings.<br> - Both view template and views defined by it have versions. Template version if defined by the version of it's owner component. View version increases with every change made to it and can be used by the orchestrator and component to determine if the async update of view was made by external links.<div><br></div><div>We will continue to flesh it out as a specification in Fuel specs repository. I will greatly appreciate any feedback on this vision, including comments, objections, concerns and questions.</div><div><br></div><div>--</div><div>Best regards,</div><div>Oleg Gelbukh</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 20, 2015 at 2:13 PM, Vladimir Kuklin <span dir="ltr"><<a href="mailto:vkuklin@mirantis.com" target="_blank">vkuklin@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Folks<div><br></div><div>Can we please stop using etherpad and move to some more usable thing as Google Docs? Etherpad seems too unusable for such discussion especially with this coloured formatting.</div><div><br></div><div>Mike</div><div><br></div><div>I currently see no need in following marketing trend for noSQL here - we need to store a set of structured data. This store should be the one that can be easily consumed directly or with some API wrapper. That is all. We will need to carefully evaluate each storage engine and decide which to pick. I personally insist on the engine that provides 100% consistency which is in fact opposite to what most of noSQL and distributed architectures provide. Nobody cares if you lose 1 billion of messages in a social network (even these messages authors) - this is almost all the time garbage with porn and cat pictures. Things will get worse if you destroy something in production serving accounting in your cloud due to the fact that nodes are </div><div><br></div><div>I agree with option #2 - we actually should have task abstraction layer with drivers for execution, but I would go with baby steps for supporting other deployment tools - currently I do not see any benefit in using Ansible for tasks that Fuel is solving. The same is almost true for containers, but this is a different story.</div><div><br></div><div>Eugene, Mike</div><div><br></div><div>I agree with you that we need to think about where to execute these serializers. I think that we could do it the following way - serializer can be executed wherever it can actually work and it should possibly put data into centralized storage for the means of logging, control and accounting. I am not sure that this is the limitation case all the users will agree with, but we need to think of it.</div><div><br></div><div>Regarding this 'last task throwing an exception issue' - we can handle this properly by simply rerunning the task that failed only due to serialization problem. Or even better - reorder its execution for later steps and try it again in a while if there are other tasks to be executed. </div><div><br></div><div>But Mike's approach of data preparation prior to deployment/workflow transaction execution seems more viable. I think, we should follow the following one: "If you do not know the data before the transaction run, this data should be calculated after this transaction ends and this data should be used for another workflow in a different transaction".</div><div><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 20, 2015 at 1:20 PM, Evgeniy L <span dir="ltr"><<a href="mailto:eli@mirantis.com" target="_blank">eli@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I have a comment regarding to when/where run translators.</div><div>I think data processing (fetching + validation + translation) should be done</div><div>as a separate stage, this way when we start deployment, we are sure</div><div>that we have everything we need to perform deployment, but I understand</div><div>that there might be exceptions and sometimes we will have to get the data on the fly.</div><div>If we are going to put translated data into some distributed store I'm not sure</div><div>if distributed approach for fetching and updating the data won't cause the problems</div><div>with race conditions, in centralised approach probability to get such problems can</div><div>be reduced.</div><div><br></div><div>Thanks,<br></div><div><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 20, 2015 at 5:14 AM, Mike Scherbakov <span dir="ltr"><<a href="mailto:mscherbakov@mirantis.com" target="_blank">mscherbakov@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Thanks Vladimir.<div>This is very important work, I'd say. I'd split it into two parts:</div><div><ol><li>Have some store where you'd dump data from serializers. Hiera should be able to read easily directly from this store.</li><li>Refactor serializers so to get rid of single python method which creates data for multiple OpenStack components, and allow deployment engineers to easily modify code of particular piece</li></ol><div>For #1, it is important to think broadly. We want this store to be used by other tools which users may have (Puppet Master, Ansible, etc.) as a source data, and so that Fuel & other tools can coexist on the same env if really needed (even though I'd ideally try to avoid it).</div></div><div>We need to have an abstraction layer there, so that we can have drivers for key-value store and for such things as Zookeeper, for instance, in the future. I think we need to address #1 in the first order, before going to #2 (if we can't do it in parallel).</div><div><br></div><div>For #2, I think we need to consider flexibility. What if we use Ansible, or container for some of our services? So we need to think where we can put these per-component / per-task serializers, so those can be used for both Puppet module & something different.</div><div><br></div><div>Also, it's interesting problem from the execution point of view. Do we run serialization on Fuel Master side or on slave nodes, where we install OpenStack? I see some issues with running it on OpenStack nodes, even though I like an idea of load distribution, etc. For instance, if you run almost all graph, and then the last task in the graph runs corresponding serializer - and there is a Python exception for whatever reason (user input leads to bug in calculation). You could get it right a way, if you tried to calculate it before overall deployment - but now you've been waiting deployment to be almost done to catch it.</div><div><br></div><div>Thank you,</div></div><br><div class="gmail_quote"><div><div><div dir="ltr">On Fri, Oct 16, 2015 at 9:22 AM Vladimir Kuklin <<a href="mailto:vkuklin@mirantis.com" target="_blank">vkuklin@mirantis.com</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div><div dir="ltr">Hey, Fuelers<div><br></div><div>TL;DR This email is about how to make </div><div><br></div><div>* Intro</div><div>I want to bring up one of the important topics on how to make Fuel more flexible. Some of you know that we have been discussing means of doing this internally and now it is time to share these thoughts with all of you.</div><div><br></div><div>As you could know per Evgeniy Li's message [0] we are looking forward splitting Fuel (specifically it's Fuel-Web) part into set of microservices each one serving their own purpose like networking configuration, partitioning, etc. </div><div><br></div><div><br></div><div>And while we are working on this it seems that we need to get rid of so-called Nailgun serializers that are put too close to business logic engine, that have a lot of duplicating attributes; you are not able to easily modify or extend them; you are not able to change their behaviour even when Fuel Library is capable of doing so - everything is hardcoded in Nailgun code without clear separation between business logic and actual deployment workflow data generation and orchestration.</div><div><br></div><div>Let me give you an example:<br></div><div><br></div><div>* Case A. Replace Linux bridges with OVS bridges by default</div><div><br></div><div>We all know that we removed OVS as much as possible from our reference architecture due to its buginess. Imagine a situation when someone magically fixed OVS and wants to use it as a provider for generic bonds and bridge. It actually means that he needs to set default provider in network_scheme for l23network puppet module to 'ovs' instead of 'lnx'. Imagine, he has put this magical OVS into a package and created a plugin. The problem here will be that he needs to override what network serializer is sending to the nodes.</div><div><br></div><div>But the problem here is that he cannot do it without editing Nailgun code or override this serializer in any way.</div><div><br></div><div>* Case B. Make Swift Partitions Known to Fuel Library</div><div><br></div><div>Imagine, you altered the way you partition your disk in Nailgun. You created a special role for swift disks which should occupy the whole disk. In this case you should be able to get this info from api and feed it to swift deployment task. But it is not so easy - this stuff is still hardcoded in deployment serializers like {mp} field of nodes array of hashes.</div><div><br></div><div>* Proposed solution</div><div><br></div><div>In order to tackle this I propose to extract these so called serializers (see links [1] and [2]) and put them closer to library. You can see that half of the code is actually duplicated for deployment and provsioning serializers and there is actually no inheritance of common code betwen them. If you want to introduce new attribute and put it into astute.yaml, you will need to rewrite Nailgun code. This is not very deployment/sysop/sysadmin engineer-friendly. Essentially, the proposal is to introduce a library of such `serializers` (I would like to call them translators actually) which could leverage inheritance, polymorphism and incapsulation pretty much in OOP mode but with ability for deployment engineers to apply versioning to serializers and allow each particular task to work with different sources of data with different versions of API.</div><div><br></div><div>What this actually means: each task has a step called 'translation' which fetches attributes from any arbitrary set of sources and converts them into the format that is consumable by the deployment stage of this task. From our current architectural point of view it will look like generation of a set of yaml files that will be merged by hiera so that each puppet task can leverage the power of hiera. </div><div><br></div><div>This actually means that in scope of our modularization initiative each module should have an API which will be accessed by those tasks in runtime right before the tasks are executed. This also means that if a user changes some of the values in the databases of those modules, rerun of such task will lead to a different result of 'translation' and trigger some actions like 'keystone_config ~> Service[keystone]' in puppet.</div><div><br></div><div>There is a tough discussion (etherpad here:[4]) on:</div><div><br></div><div>1) how to handle versioning/revert capabilities</div><div>2) where to store output produced by those 'translators'</div><div>3) which type of the storage to use</div><div><br></div><div>Please, feel free to provide your feedback on this approach and tell me where this approach is going to be wrong.</div><div><br></div><div>[0] <a href="http://permalink.gmane.org/gmane.comp.cloud.openstack.devel/66563" target="_blank">http://permalink.gmane.org/gmane.comp.cloud.openstack.devel/66563</a><br></div><div>[1] <a href="https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/orchestrator/deployment_serializers.py" target="_blank">https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/orchestrator/deployment_serializers.py</a></div><div>[2] <a href="https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/orchestrator/provisioning_serializers.py" target="_blank">https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/orchestrator/provisioning_serializers.py</a></div><div>[3] <a href="https://github.com/xenolog/l23network" target="_blank">https://github.com/xenolog/l23network</a></div><div>[4] <a href="https://etherpad.openstack.org/p/data-processor-per-component" target="_blank">https://etherpad.openstack.org/p/data-processor-per-component</a><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div><div dir="ltr">Yours Faithfully,<br>Vladimir Kuklin,<br>Fuel Library Tech Lead,<br>Mirantis, Inc.<br>+7 (495) 640-49-04<br>+7 (926) 702-39-68<br>Skype kuklinvv<br>35bk3, Vorontsovskaya Str.<br>Moscow, Russia,<br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.com</a><br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.ru</a><br><a href="mailto:vkuklin@mirantis.com" target="_blank">vkuklin@mirantis.com</a></div></div></div></div>
</div></div></div></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><span><font color="#888888"><br>
</font></span></blockquote></div><span><font color="#888888"><div dir="ltr">-- <br></div><div dir="ltr">Mike Scherbakov<br>#mihgen</div>
</font></span><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div><div dir="ltr">Yours Faithfully,<br>Vladimir Kuklin,<br>Fuel Library Tech Lead,<br>Mirantis, Inc.<br>+7 (495) 640-49-04<br>+7 (926) 702-39-68<br>Skype kuklinvv<br>35bk3, Vorontsovskaya Str.<br>Moscow, Russia,<br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.com</a><br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.ru</a><br><a href="mailto:vkuklin@mirantis.com" target="_blank">vkuklin@mirantis.com</a></div></div></div></div>
</div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div><div dir="ltr">Yours Faithfully,<br>Vladimir Kuklin,<br>Fuel Library Tech Lead,<br>Mirantis, Inc.<br>+7 (495) 640-49-04<br>+7 (926) 702-39-68<br>Skype kuklinvv<br>35bk3, Vorontsovskaya Str.<br>Moscow, Russia,<br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.com</a><br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.ru</a><br><a href="mailto:vkuklin@mirantis.com" target="_blank">vkuklin@mirantis.com</a></div></div></div></div>
</div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div></div>