<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I am in favour of ditching or at least refactoring devstack because during the last year I often found myself blocked from fixing some zuul/jobs issues because the buggy code was still required by legacy devstack jobs that nobody had time maintain or fix, so they were isolated and the default job configurations were forced to use dirty hack needed for keeping these working.<div class=""><br class=""></div><div class="">One such example is that there is a task that does a "chmod -R 0777 -R" on the entire source tree, a total security threat. </div><div class=""><br class=""></div><div class="">In order to make other jobs running correctly* I had to rely undoing the damage done by such chmod because I was not able to disable the historical hack.<br class=""><div><br class=""></div><div>* ansible throws warning with unsafe file permissions</div><div>* ssh refuses to load unsafe keys</div><div><br class=""></div><div>That is why I am in favor of dropping features that are slowing down the progress of others.</div><div><br class=""></div><div>I know that the reality is more complicated but I also think that sometimes less* is more. </div><div><br class=""></div><div><br class=""></div><div>* deployment projects ;)</div><div><br class=""><blockquote type="cite" class=""><div class="">On 4 Jun 2019, at 04:36, Dean Troyer <<a href="mailto:dtroyer@gmail.com" class="">dtroyer@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class=""><br class=""><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 3 Jun 2019, 15:59 Clark Boylan, <<a href="mailto:cboylan@sapwetik.org" class="">cboylan@sapwetik.org</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">On Sat, Jun 1, 2019, at 5:36 AM, Mohammed Naser wrote:<br class="">> Hi everyone,<br class="">><span class="Apple-converted-space"> </span><br class="">> This is something that I've discussed with a few people over time and<br class="">> I think I'd probably want to bring it up by now. I'd like to propose<br class="">> and ask if it makes sense to perhaps replace devstack entirely with<br class="">> openstack-ansible. I think I have quite a few compelling reasons to<br class="">> do this that I'd like to outline, as well as why I *feel* (and I could<br class="">> be biased here, so call me out!) that OSA is the best option in terms<br class="">> of a 'replacement'<br class="">><span class="Apple-converted-space"> </span><br class="">> # Why not another deployment project?<br class="">> I actually thought about this part too and considered this mainly for<br class="">> ease of use for a *developer*.<br class="">><span class="Apple-converted-space"> </span><br class="">> At this point, Puppet-OpenStack pretty much only deploys packages<br class="">> (which means that it has no build infrastructure, a developer can't<br class="">> just get $commit checked out and deployed).<br class="">><span class="Apple-converted-space"> </span><br class="">> TripleO uses Kolla containers AFAIK and those have to be pre-built<br class="">> beforehand, also, I feel they are much harder to use as a developer<br class="">> because if you want to make quick edits and restart services, you have<br class="">> to enter a container and make the edit there and somehow restart the<br class="">> service without the container going back to it's original state.<br class="">> Kolla-Ansible and the other combinations also suffer from the same<br class="">> "issue".<br class="">><span class="Apple-converted-space"> </span><br class="">> OpenStack Ansible is unique in the way that it pretty much just builds<br class="">> a virtualenv and installs packages inside of it. The services are<br class="">> deployed as systemd units. This is very much similar to the current<br class="">> state of devstack at the moment (minus the virtualenv part, afaik).<br class="">> It makes it pretty straight forward to go and edit code if you<br class="">> need/have to. We also have support for Debian, CentOS, Ubuntu and<br class="">> SUSE. This allows "devstack 2.0" to have far more coverage and make<br class="">> it much more easy to deploy on a wider variety of operating systems.<br class="">> It also has the ability to use commits checked out from Zuul so all<br class="">> the fancy Depends-On stuff we use works.<br class="">><span class="Apple-converted-space"> </span><br class="">> # Why do we care about this, I like my bash scripts!<br class="">> As someone who's been around for a *really* long time in OpenStack,<br class="">> I've seen a whole lot of really weird issues surface from the usage of<br class="">> DevStack to do CI gating. For example, one of the recent things is<br class="">> the fact it relies on installing package-shipped noVNC, where as the<br class="">> 'master' noVNC has actually changed behavior a few months back and it<br class="">> is completely incompatible at this point (it's just a ticking thing<br class="">> until we realize we're entirely broken).<br class=""><br class="">I'm not sure this is a great example case. We consume prebuilt software for many of our dependencies. Everything from the kernel to the database to rabbitmq to ovs (and so on) are consumed as prebuilt packages from our distros. In many cases this is desirable to ensure that our software work with the other software out there in the wild that people will be deploying with.<br class=""><br class="">><span class="Apple-converted-space"> </span><br class="">> To this day, I still see people who want to POC something up with<br class="">> OpenStack or *ACTUALLY* try to run OpenStack with DevStack. No matter<br class="">> how many warnings we'll put up, they'll always try to do it. With<br class="">> this way, at least they'll have something that has the shape of an<br class="">> actual real deployment. In addition, it would be *good* in the<br class="">> overall scheme of things for a deployment system to test against,<br class="">> because this would make sure things don't break in both ways.<br class="">><span class="Apple-converted-space"> </span><br class="">> Also: we run Zuul for our CI which supports Ansible natively, this can<br class="">> remove one layer of indirection (Zuul to run Bash) and have Zuul run<br class="">> the playbooks directly from the executor.<br class=""><br class="">I think if you have developers running a small wrapper locally to deploy this new development stack you should run that same wrapper in CI. This ensure the wrapper doesn't break.<br class=""><br class="">><span class="Apple-converted-space"> </span><br class="">> # So how could we do this?<br class="">> The OpenStack Ansible project is made of many roles that are all<br class="">> composable, therefore, you can think of it as a combination of both<br class="">> Puppet-OpenStack and TripleO (back then). Puppet-OpenStack contained<br class="">> the base modules (i.e. puppet-nova, etc) and TripleO was the<br class="">> integration of all of it in a distribution. OSA is currently both,<br class="">> but it also includes both Ansible roles and playbooks.<br class="">><span class="Apple-converted-space"> </span><br class="">> In order to make sure we maintain as much of backwards compatibility<br class="">> as possible, we can simply run a small script which does a mapping of<br class="">> devstack => OSA variables to make sure that the service is shipped<br class="">> with all the necessary features as per local.conf.<br class="">><span class="Apple-converted-space"> </span><br class="">> So the new process could be:<br class="">><span class="Apple-converted-space"> </span><br class="">> 1) parse local.conf and generate Ansible variables files<br class="">> 2) install Ansible (if not running in gate)<br class="">> 3) run playbooks using variable generated in #1<br class="">><span class="Apple-converted-space"> </span><br class="">> The neat thing is after all of this, devstack just becomes a thin<br class="">> wrapper around Ansible roles. I also think it brings a lot of hands<br class="">> together, involving both the QA team and OSA team together, which I<br class="">> believe that pooling our resources will greatly help in being able to<br class="">> get more done and avoiding duplicating our efforts.<br class="">><span class="Apple-converted-space"> </span><br class="">> # Conclusion<br class="">> This is a start of a very open ended discussion, I'm sure there is a<br class="">> lot of details involved here in the implementation that will surface,<br class="">> but I think it could be a good step overall in simplifying our CI and<br class="">> adding more coverage for real potential deployers. It will help two<br class="">> teams unite together and have more resources for something (that<br class="">> essentially is somewhat of duplicated effort at the moment).<br class="">><span class="Apple-converted-space"> </span><br class="">> I will try to pick up sometime to POC a simple service being deployed<br class="">> by an OSA role instead of Bash, placement which seems like a very<br class="">> simple one and share that eventually.<br class="">><span class="Apple-converted-space"> </span><br class="">> Thoughts? :)<br class=""><br class="">For me there are two major items to consider that haven't been brought up yet. The first is devstack's (lack of) speed. Any replacement should be at least as quick as the current tooling because the current tooling is slow enough already.</blockquote></div></div><div dir="auto" class=""><br class=""></div><div dir="auto" class=""></div><div dir="auto" class="">This is important. We would need to see benchmark comparisons between a devstack install and an OSA install. Shell may be slow but Ansible is generally slower. That's fine in production when reliability is king, but we need fast iteration for development.</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">I haven't looked under the covers of devstack for some time, but it previously installed all python deps in one place, whereas OSA has virtualenvs for each service which could take a while to build. Perhaps this is configurable.</div><div dir="auto" class=""><br class=""></div><div dir="auto" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">The other is logging. I spend a lot of time helping people to debug CI job runs and devstack has grown a fairly effective set of logging that just about any time I have to help debug another deployment tool's CI jobs I miss (because they tend to log only a tiny fraction of what devstack logs).<br class=""><br class="">Clark</blockquote></div></div></div></div></blockquote></div><br class=""></div></body></html>