<div dir="ltr">Dmitry,<div><br></div><div>Q1. Yes.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-size:12.8000001907349px">where do you plan to actually perform settings manipulation? </span></blockquote><div>It was one of the critical blockers. Most of the settings are baked inside</div><div>fuel-library. Your feature [1] partially fixes this BTW. Which is good.</div><div>Partially, because only limited number of tasks has defined overrides.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-size:12.8000001907349px">scheduled basis run nailgun-cm-agent</span></blockquote><div>Currently i see better way. nailgun-cm-agent (or whatever) should just check<br></div><div>system status (i.e. puppet apply --noop) and report back. User will decide</div><div>apply changes or not.</div><div><br></div><div>Q2.</div><div>Yes, i did. One more use case covered. Please see first table in [2].</div><div><br></div><div>Q4. Agree. Here is the bug [3]</div><div><br></div><div>Q3,Q5, Q6</div><div>Good.</div><div><br></div><div>[1] <a href="https://blueprints.launchpad.net/fuel/+spec/openstack-config-change">https://blueprints.launchpad.net/fuel/+spec/openstack-config-change</a></div><div>[2] <a href="https://docs.google.com/document/d/1bVVZJR73pBWB_WbfOzC-fh84pVsi20v4n3rvn38F4OI/edit">https://docs.google.com/document/d/1bVVZJR73pBWB_WbfOzC-fh84pVsi20v4n3rvn38F4OI/edit</a> (limited access)</div><div>[3] <a href="https://bugs.launchpad.net/fuel/+bug/1525872">https://bugs.launchpad.net/fuel/+bug/1525872</a></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 14, 2015 at 4:39 AM, Dmitriy Novakovskiy <span dir="ltr"><<a href="mailto:dnovakovskiy@mirantis.com" target="_blank">dnovakovskiy@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Roman,<div><br></div><div>Thanks a lot for the feedback. We'll be planning improvements for [1] in upcoming 9.0 cycle, so your input and this discussion are very helpful and much appreciated.</div><div><br></div><div><font color="#000000">In overall, the concept for nailgun-cm-agent looks interesting, but I think you'll face some problems with it:</font></div><div><font color="#000000">- idempotency of puppet modules</font></div><div><font color="#000000">- lack of exposed parameters (fuel-lib hacking)</font></div><div><font color="#000000">- speed of re-runs of configuration mgmt (that we're already working on in 8.0)</font></div><div><br></div><div>Now, my comments and questions.</div><div><br></div><div><b>1) nailgun-cm-agent concept</b></div><div>Q1. Do I understand correctly that the planned UX is:</div><div>- Allow user to change configuration as dictated by Fuel (btw, where do you plan to actually perform settings manipulation? Directly in Puppet modules/manifests?)</div><div>- On scheduled basis run nailgun-cm-agent and let it bring overall system state to be consistent with latest changes</div><div>?</div><div><br></div><div><b>2) "Advanced settings" [1] feature feedback</b></div><div>Q2. Please share the details about 13 real world tasks that you used for testing. Have you had a chance to test this same list against [1], as you did with fuel-cm-agent approach? I need to know what from real world is doable and what not with current state of [1]</div><div>Q3. "<span style="font-size:13px">It allows just apply, not track changes</span>" - that's true, 8.0 has first MVP of this feature in place, and we don't yet have much tracking capability (other than looking at logs in DB, when what config change yaml was uploaded). We will be improving it in 9.0 cycle</div><div>Q4. "<span style="font-size:13px">Moreover works weird, </span><span style="font-size:13px">if multiple changes uploaded, applying not the latest, but initial </span><span style="font-size:13px">config change.</span>" - can you please share the detailed example? I'm not sure I understood it, but so far sounds like a bug that needs to be fixed.</div><div>Q5. "<span style="font-size:13px">Just limited number[1] of resources/tasks has support.</span>" - this is the limitation of what configs are shipped out of the box. When 8.0 is released, we'll have a documented way to add support for any OpenStack config file that Fuel tasks can reach</div><div>Q6. "<span style="font-size:13px">Can we start moving all (non orchestrating) data into CMDB? yaml under git </span><span style="font-size:13px">or any existing solution." We're now discussing major refactoring effort to be done in Fuel to integrate with Solar and solve some of the long standing </span></div><div><br></div><div><span style="font-size:13px">[1] </span><a href="https://blueprints.launchpad.net/fuel/+spec/openstack-config-change" style="font-size:13px" target="_blank">https://blueprints.launchpad.net/fuel/+spec/openstack-config-change</a><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 11, 2015 at 6:21 PM, Roman Sokolkov <span dir="ltr"><<a href="mailto:rsokolkov@mirantis.com" target="_blank">rsokolkov@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Oleg,<div><br></div><div>thanks. I've tried it [1], looks like it works.<div><br></div><div>- GOOD. "override_resource" resource. Like "back door" into puppet modules.</div><div>- BAD. It allows just apply, not track changes. Moreover works weird,</div><div>if multiple changes uploaded, applying not the latest, but initial</div><div>config change.</div><div>- BAD. Just limited number[1] of resources/tasks has support.</div><div><br></div><div>BTW, my feeling that we should NOT develop this approach in the same way.</div><div><br></div><div>I'm not an expert, but as long-term</div><div>- Can we start moving all (non orchestrating) data into CMDB? yaml under git</div><div>or any existing solution.</div><div>- Can we track nodes state? For example, start by cron all puppet tasks with --noop option</div><div>and check puppet state. Then if "out of sync" node start blinking YELLOW and user</div><div>can push button, if needed.</div><div><br></div><div>Thanks</div><div><br></div><div>[1] <a href="https://blueprints.launchpad.net/fuel/+spec/openstack-config-change" target="_blank">https://blueprints.launchpad.net/fuel/+spec/openstack-config-change</a></div><div>[2] <a href="http://paste.openstack.org/show/481677/" target="_blank">http://paste.openstack.org/show/481677/</a></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 11, 2015 at 4:34 PM, Oleg Gelbukh <span dir="ltr"><<a href="mailto:ogelbukh@mirantis.com" target="_blank">ogelbukh@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Roman,<div><br></div><div>Changing arbitrary parameters supported by respective Puppet manifests for OpenStack services is implemented in this blueprint [1]. It is being landed in release 8.0.</div><div><br></div><div>[1] <a href="https://blueprints.launchpad.net/fuel/+spec/openstack-config-change" target="_blank">https://blueprints.launchpad.net/fuel/+spec/openstack-config-change</a></div><span><div><br></div><div>--</div><div>Best regards,</div><div>Oleg Gelbukh</div></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 3, 2015 at 5:28 PM, Roman Sokolkov <span dir="ltr"><<a href="mailto:rsokolkov@mirantis.com" target="_blank">rsokolkov@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Folks,<div><br></div><div>little bit more research done in regards #2 usability.</div><div><br></div><div>I've selected 13 real-world tasks from customer (i.e. update flag X in nova.conf):</div><div>- 6/13 require fuel-library patching (or is #2 unusable)</div><div>- 3/13 are OK and can be done with #2</div><div>- 4/13 can be done with some limitations.</div><div><br></div><div>If needed i'll provide details.</div><div><br></div><div>Rough statistics is that <b>only ~20-25% of use cases can be done with #2</b>.</div><div><br></div><div>Let me give a very popular use case that will fail with #2. Assume we'r executing whole task graph every two hours.</div><div>We want to change nova.conf "DEFAULT/amqp_durable_queues" from False to True.</div><div><br></div><div>There is no parameter in hiera for "amqp_durable_queues". We have two solutions here (both are bad):</div><div>1) Redefine "DEFAULT/amqp_durable_queues" = True in plugin task. What will happen on the node. amqp_durable_queues will continue changing value between True and False on every execution. We shouldn't do it this way.</div><div>2) Patch fuel-library. Value for amqp_durable_queues should be taken from hiera. This is also one way ticket.</div><div><br></div><div>Thanks</div><div><br></div><div><br></div><div><br></div><div><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 3, 2015 at 11:28 AM, Oleg Gelbukh <span dir="ltr"><<a href="mailto:ogelbukh@mirantis.com" target="_blank">ogelbukh@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Roman,<div><br></div><div>Thank you. This is great research. </div><div class="gmail_extra"><br></div><div class="gmail_extra">Could we have a conversation to discuss this? I'm especially interested in idempotency problems of the fuel-library modules and the common way to provide serialised data to the deployment.</div><div class="gmail_extra"><br></div><div class="gmail_extra">--</div><div class="gmail_extra">Best regards,</div><div class="gmail_extra">Oleg Gelbukh</div><div class="gmail_extra">Mirantis Inc</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Tue, Dec 1, 2015 at 6:38 PM, Roman Sokolkov <span dir="ltr"><<a href="mailto:rsokolkov@mirantis.com" target="_blank">rsokolkov@mirantis.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hello, folks.<div><br></div><div>We need any kind of CM for Fuel 7.0. Otherwise new project with 800+ nodes</div><div>will be near impossible to support. Customer always wants to change something.</div><div><br></div><div>In our opinion, there are two major approaches for CM:</div><div><br></div><div>#1 Independent CM (Puppet master, Chef, Ansible, whatever)</div><div>#2 Fuel-based CM</div><div><br></div><div>Solution for #2</div><div>----------</div><div><br></div><div>Fuel has all info about configuration. So we've tried to</div><div>unlock "Settings" [0] and push "deploy" button.</div><div><br></div><div>Major findings:</div><div><br></div><div>* Task idem-potency. Looks like most of the tasks are idempotent.</div><div>We've skipped 3 tasks on controller and were able to get NO downtime</div><div>for Horizon and "nova list". BTW deeper QA required.</div><div><br></div><div>* Standard changes. Operator can change parameters via WebUI, CLI or API.<br></div><div>For example, i was able to deploy Sahara. Unfortunately there is not foolproof.</div><div>I mean some changes can lead to broken cloud...</div><div><br></div><div>* Non-standard changes. Any other changes can be done with plugins.</div><div>We can modify plugin tasks and scripts (all except UI flags). And then just</div><div>do "--update" + "--sync". BTW, we can change UI for particular env via API</div><div>by modifying "clusters/X/attributes".<br><br>Conclusion</div><div>----------</div><div><br></div><div>- This works (We have service under cron that runs tasks) [1]</div><div>- NOT ready for production (in current state)<br></div><div>- This requires much deeper testing</div><div><br></div><div><br></div><div>I want to hear thoughts about approach above?</div><div>What is the current status/plans for CM? I saw this discussion [2]</div><div><br></div><div>References</div><div>----------</div><div><br></div><div>[0] <a href="https://github.com/rsokolkov/fuel-web/commit/366daaa2eb874c8e54c2d39be475223937cd317d" target="_blank">https://github.com/rsokolkov/fuel-web/commit/366daaa2eb874c8e54c2d39be475223937cd317d</a></div><div>[1] <a href="https://docs.google.com/presentation/d/12kkh1hu4ZrY9S6XXsY_HWaesFwESfxbl5czUwde8isM/edit#slide=id.p" target="_blank">https://docs.google.com/presentation/d/12kkh1hu4ZrY9S6XXsY_HWaesFwESfxbl5czUwde8isM/edit#slide=id.p</a><br></div><div>[2] <a href="https://etherpad.openstack.org/p/lcm-use-cases" target="_blank">https://etherpad.openstack.org/p/lcm-use-cases</a><span><font color="#888888"><br></font></span></div><span><font color="#888888"><div><br></div><div>-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Roman Sokolkov,<div>Deployment Engineer,</div><div>Mirantis, Inc.<br>Skype rsokolkov,<br><a href="mailto:rsokolkov@mirantis.com" target="_blank">rsokolkov@mirantis.com</a></div></div></div></div></div></div></div>
</div></font></span></div>
<br></div></div><span>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></span></blockquote></div><br></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Roman Sokolkov,<div>Deployment Engineer,</div><div>Mirantis, Inc.<br>Skype rsokolkov,<br><a href="mailto:rsokolkov@mirantis.com" target="_blank">rsokolkov@mirantis.com</a></div></div></div></div></div></div></div>
</div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Roman Sokolkov,<div>Deployment Engineer,</div><div>Mirantis, Inc.<br>Skype rsokolkov,<br><a href="mailto:rsokolkov@mirantis.com" target="_blank">rsokolkov@mirantis.com</a></div></div></div></div></div></div></div>
</font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Roman Sokolkov,<div>Deployment Engineer,</div><div>Mirantis, Inc.<br>Skype rsokolkov,<br><a href="mailto:rsokolkov@mirantis.com" target="_blank">rsokolkov@mirantis.com</a></div></div></div></div></div></div></div>
</div>