<div dir="ltr">Hi, Roman<div><br></div><div>So here is how we think we are going to tackle this. The whole idea is to make Fuel architecture much more flexible. We are actively working on this and are going to present draft of initial documents pretty shortly before the end of the year.</div><div><br></div><div><br></div><div>* <b>Short (very-very short) summary of our plan</b></div><div><br></div><div><b>** Nailgun will be split into pieces</b><br></div><div><b><br></b></div><div>*** <b>Managers split-off</b></div><div>Business logic things such as network allocation, volume allocation and so on are going to become services with their own APIs</div><div><br></div><div><b>*** Serializers externalization and pluggability</b></div><div><b><br></b></div><div>Current serializers are going to become separate entities that will be pluggable and easily extendable. These serializers will be able to consume data from any service you register in Nailgun as a data source. These data sources could be: </div><div><br></div><div>1) Nailgun network/volume/openstack_cluster managers</div><div>2) any 3rd party service with an API:</div><div> a) LDAP</div><div> b) Inventory</div><div> c) external puppet master</div><div> d) etc.</div><div><br></div><div><b>** There is going to be a configuration storage</b></div><div><b><br></b></div><div>All the stuff created by serializers is going to be stored in a database in versioned and hierarchical format.</div><div>This storage will be a source for deployment tasks execution.</div><div><br></div><div><b>** Difference in configuration is going to be handled by new components</b></div><div><br></div><div>Right now we have some PoC code to make it happen and once it reaches working stage, we will upload it to upstream and continue working on it in open format. It is going to consume the difference between cluster configuration and orchestrate execution of particular deployment tasks from Fuel Library or any other 'runnable' resource wrapped into its format, e.g. container, ansible playbook, whatever. </div><div><br></div><div><b>* How it is all related to you problem</b></div><div><b><br></b></div><div>In this case you should be able to proof-test this approach - you could take any simple DB and make current serializers store data into this DB. This will allow you to fetch additional data from other sources and write also to this config DB. E.g. if you want to fetch some data from puppet master - do the following:</div><div>1) send a request to puppet master to compile the catalogue for a particular node</div><div>2) parse this catalogue and extract required things into config DB</div><div>3) run deployment with new values from config db</div><div><br></div><div><b>* Existing feature tackling this issue</b></div><div><b><br></b></div><div>BTW, there is already a feature which partially addresses your demands: <a href="https://blueprints.launchpad.net/fuel/+spec/openstack-config-change" target="_blank">https://blueprints.launchpad.net/fuel/+spec/openstack-config-change</a></div><div><b><br></b></div><div>*<b> Idempotency fixes</b></div><div><b><br></b></div><div>Well, if you find any problems - please file bugs. There are some execs in puppet code which make tasks not 100% idempotent, but these execs are actually harmless. If you find any other issues with idempotency - please, file bugs for them. </div><div><br></div><div><b>* Issues like '</b><span style="font-size:12.8000001907349px"><b>amqp_durable_queues</b></span><b>' redefinition</b></div><div><b><br></b></div><div>These issues are going to be tackled by strict responsibilities distribution between 'serializers' part and deployment tasks part.</div><div><br></div><div>E.g. there should be no more than 1 task doing one thing. '<span style="font-size:12.8000001907349px">amqp_durable_queues' should be configured exactly with one task, not with 2 tasks.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">To achieve this the plugin you mentioned should only change the value of '</span><span style="font-size:12.8000001907349px">amqp_durable_queues</span><span style="font-size:12.8000001907349px">' in the config DB, while the regular task that will be called will be the one you have in the core of Fuel Library.</span></div><div><br></div><div><span style="font-size:12.8000001907349px"><b>* Issues with deployment time</b></span></div><div><span style="font-size:12.8000001907349px"><b><br></b></span></div><div><span style="font-size:12.8000001907349px">We are working hard on introducing improved orchestration in experimental mode in 8.0 - please check out <a href="https://blueprints.launchpad.net/fuel/+spec/task-based-deployment-astute" target="_blank">https://blueprints.launchpad.net/fuel/+spec/task-based-deployment-astute</a></span></div><div><b><br></b></div><div>Hope, my answer was not too long. Feel free to ask questions.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 3, 2015 at 5:28 PM, Roman Sokolkov <span dir="ltr"><<a href="mailto:rsokolkov@mirantis.com" target="_blank">rsokolkov@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Folks,<div><br></div><div>little bit more research done in regards #2 usability.</div><div><br></div><div>I've selected 13 real-world tasks from customer (i.e. update flag X in nova.conf):</div><div>- 6/13 require fuel-library patching (or is #2 unusable)</div><div>- 3/13 are OK and can be done with #2</div><div>- 4/13 can be done with some limitations.</div><div><br></div><div>If needed i'll provide details.</div><div><br></div><div>Rough statistics is that <b>only ~20-25% of use cases can be done with #2</b>.</div><div><br></div><div>Let me give a very popular use case that will fail with #2. Assume we'r executing whole task graph every two hours.</div><div>We want to change nova.conf "DEFAULT/amqp_durable_queues" from False to True.</div><div><br></div><div>There is no parameter in hiera for "amqp_durable_queues". We have two solutions here (both are bad):</div><div>1) Redefine "DEFAULT/amqp_durable_queues" = True in plugin task. What will happen on the node. amqp_durable_queues will continue changing value between True and False on every execution. We shouldn't do it this way.</div><div>2) Patch fuel-library. Value for amqp_durable_queues should be taken from hiera. This is also one way ticket.</div><div><br></div><div>Thanks</div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 3, 2015 at 11:28 AM, Oleg Gelbukh <span dir="ltr"><<a href="mailto:ogelbukh@mirantis.com" target="_blank">ogelbukh@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Roman,<div><br></div><div>Thank you. This is great research. </div><div class="gmail_extra"><br></div><div class="gmail_extra">Could we have a conversation to discuss this? I'm especially interested in idempotency problems of the fuel-library modules and the common way to provide serialised data to the deployment.</div><div class="gmail_extra"><br></div><div class="gmail_extra">--</div><div class="gmail_extra">Best regards,</div><div class="gmail_extra">Oleg Gelbukh</div><div class="gmail_extra">Mirantis Inc</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Tue, Dec 1, 2015 at 6:38 PM, Roman Sokolkov <span dir="ltr"><<a href="mailto:rsokolkov@mirantis.com" target="_blank">rsokolkov@mirantis.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hello, folks.<div><br></div><div>We need any kind of CM for Fuel 7.0. Otherwise new project with 800+ nodes</div><div>will be near impossible to support. Customer always wants to change something.</div><div><br></div><div>In our opinion, there are two major approaches for CM:</div><div><br></div><div>#1 Independent CM (Puppet master, Chef, Ansible, whatever)</div><div>#2 Fuel-based CM</div><div><br></div><div>Solution for #2</div><div>----------</div><div><br></div><div>Fuel has all info about configuration. So we've tried to</div><div>unlock "Settings" [0] and push "deploy" button.</div><div><br></div><div>Major findings:</div><div><br></div><div>* Task idem-potency. Looks like most of the tasks are idempotent.</div><div>We've skipped 3 tasks on controller and were able to get NO downtime</div><div>for Horizon and "nova list". BTW deeper QA required.</div><div><br></div><div>* Standard changes. Operator can change parameters via WebUI, CLI or API.<br></div><div>For example, i was able to deploy Sahara. Unfortunately there is not foolproof.</div><div>I mean some changes can lead to broken cloud...</div><div><br></div><div>* Non-standard changes. Any other changes can be done with plugins.</div><div>We can modify plugin tasks and scripts (all except UI flags). And then just</div><div>do "--update" + "--sync". BTW, we can change UI for particular env via API</div><div>by modifying "clusters/X/attributes".<br><br>Conclusion</div><div>----------</div><div><br></div><div>- This works (We have service under cron that runs tasks) [1]</div><div>- NOT ready for production (in current state)<br></div><div>- This requires much deeper testing</div><div><br></div><div><br></div><div>I want to hear thoughts about approach above?</div><div>What is the current status/plans for CM? I saw this discussion [2]</div><div><br></div><div>References</div><div>----------</div><div><br></div><div>[0] <a href="https://github.com/rsokolkov/fuel-web/commit/366daaa2eb874c8e54c2d39be475223937cd317d" target="_blank">https://github.com/rsokolkov/fuel-web/commit/366daaa2eb874c8e54c2d39be475223937cd317d</a></div><div>[1] <a href="https://docs.google.com/presentation/d/12kkh1hu4ZrY9S6XXsY_HWaesFwESfxbl5czUwde8isM/edit#slide=id.p" target="_blank">https://docs.google.com/presentation/d/12kkh1hu4ZrY9S6XXsY_HWaesFwESfxbl5czUwde8isM/edit#slide=id.p</a><br></div><div>[2] <a href="https://etherpad.openstack.org/p/lcm-use-cases" target="_blank">https://etherpad.openstack.org/p/lcm-use-cases</a><span><font color="#888888"><br></font></span></div><span><font color="#888888"><div><br></div><div>-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Roman Sokolkov,<div>Deployment Engineer,</div><div>Mirantis, Inc.<br>Skype rsokolkov,<br><a href="mailto:rsokolkov@mirantis.com" target="_blank">rsokolkov@mirantis.com</a></div></div></div></div></div></div></div>
</div></font></span></div>
<br></div></div><span>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></span></blockquote></div><br></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Roman Sokolkov,<div>Deployment Engineer,</div><div>Mirantis, Inc.<br>Skype rsokolkov,<br><a href="mailto:rsokolkov@mirantis.com" target="_blank">rsokolkov@mirantis.com</a></div></div></div></div></div></div></div>
</div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr">Yours Faithfully,<br>Vladimir Kuklin,<br>Fuel Library Tech Lead,<br>Mirantis, Inc.<br>+7 (495) 640-49-04<br>+7 (926) 702-39-68<br>Skype kuklinvv<br>35bk3, Vorontsovskaya Str.<br>Moscow, Russia,<br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.com</a><br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.ru</a><br><a href="mailto:vkuklin@mirantis.com" target="_blank">vkuklin@mirantis.com</a></div></div></div></div>
</div>