Open Stack

On Tue, 2019-06-04 at 08:56 +0100, Sorin Sbarnea wrote:
> I am in favour of ditching or at least refactoring devstack because during the last year I often found myself blocked
> from fixing some zuul/jobs issues because the buggy code was still required by legacy devstack jobs that nobody had
> time maintain or fix, so they were isolated and the default job configurations were forced to use dirty hack needed
> for keeping these working.

this sound like the issue is more realted to the fact that it is still useing a legacy job.
why not move it over to the ansible native devstack jobs.

> 
> One such example is that there is a task that does a "chmod -R 0777 -R" on the entire source tree, a total security
> threat. 
in a ci env it is not. and in a development env if it was in devstack gate or in the ansible jobs it is not.
i would not want this in a production system but it feels a little contived.
> 
> In order to make other jobs running correctly* I had to rely undoing the damage done by such chmod because I was not
> able to disable the historical hack.
> 
> * ansible throws warning with unsafe file permissions
> * ssh refuses to load unsafe keys
> 
> That is why I am in favor of dropping features that are slowing down the progress of others.
that is a self contracdicting statement. if i depend on a feature then droping it slows donw my progress.
e.g. if you state that as a goal you will find you will almost always fail as to speed someone up you slow
someone else down. what you want to aim for is a better solution that supports both usecase in a clean and defiend way.
> 
> I know that the reality is more complicated but I also think that sometimes less* is more. 
> 
> 
> * deployment projects ;)
> 
> > On 4 Jun 2019, at 04:36, Dean Troyer <dtroyer at gmail.com> wrote:
> > 
> > 
> > 
> > On Mon, 3 Jun 2019, 15:59 Clark Boylan, <cboylan at sapwetik.org <mailto:cboylan at sapwetik.org>> wrote:
> > On Sat, Jun 1, 2019, at 5:36 AM, Mohammed Naser wrote:
> > > Hi everyone,
> > > 
> > > This is something that I've discussed with a few people over time and
> > > I think I'd probably want to bring it up by now.  I'd like to propose
> > > and ask if it makes sense to perhaps replace devstack entirely with
> > > openstack-ansible.  I think I have quite a few compelling reasons to
> > > do this that I'd like to outline, as well as why I *feel* (and I could
> > > be biased here, so call me out!) that OSA is the best option in terms
> > > of a 'replacement'
> > > 
> > > # Why not another deployment project?
> > > I actually thought about this part too and considered this mainly for
> > > ease of use for a *developer*.
> > > 
> > > At this point, Puppet-OpenStack pretty much only deploys packages
> > > (which means that it has no build infrastructure, a developer can't
> > > just get $commit checked out and deployed).
> > > 
> > > TripleO uses Kolla containers AFAIK and those have to be pre-built
> > > beforehand, also, I feel they are much harder to use as a developer
> > > because if you want to make quick edits and restart services, you have
> > > to enter a container and make the edit there and somehow restart the
> > > service without the container going back to it's original state.
> > > Kolla-Ansible and the other combinations also suffer from the same
> > > "issue".
> > > 
> > > OpenStack Ansible is unique in the way that it pretty much just builds
> > > a virtualenv and installs packages inside of it.  The services are
> > > deployed as systemd units.  This is very much similar to the current
> > > state of devstack at the moment (minus the virtualenv part, afaik).
> > > It makes it pretty straight forward to go and edit code if you
> > > need/have to.  We also have support for Debian, CentOS, Ubuntu and
> > > SUSE.  This allows "devstack 2.0" to have far more coverage and make
> > > it much more easy to deploy on a wider variety of operating systems.
> > > It also has the ability to use commits checked out from Zuul so all
> > > the fancy Depends-On stuff we use works.
> > > 
> > > # Why do we care about this, I like my bash scripts!
> > > As someone who's been around for a *really* long time in OpenStack,
> > > I've seen a whole lot of really weird issues surface from the usage of
> > > DevStack to do CI gating.  For example, one of the recent things is
> > > the fact it relies on installing package-shipped noVNC, where as the
> > > 'master' noVNC has actually changed behavior a few months back and it
> > > is completely incompatible at this point (it's just a ticking thing
> > > until we realize we're entirely broken).
> > 
> > I'm not sure this is a great example case. We consume prebuilt software for many of our dependencies. Everything
> > from the kernel to the database to rabbitmq to ovs (and so on) are consumed as prebuilt packages from our distros.
> > In many cases this is desirable to ensure that our software work with the other software out there in the wild that
> > people will be deploying with.
> > 
> > > 
> > > To this day, I still see people who want to POC something up with
> > > OpenStack or *ACTUALLY* try to run OpenStack with DevStack.  No matter
> > > how many warnings we'll put up, they'll always try to do it.  With
> > > this way, at least they'll have something that has the shape of an
> > > actual real deployment.  In addition, it would be *good* in the
> > > overall scheme of things for a deployment system to test against,
> > > because this would make sure things don't break in both ways.
> > > 
> > > Also: we run Zuul for our CI which supports Ansible natively, this can
> > > remove one layer of indirection (Zuul to run Bash) and have Zuul run
> > > the playbooks directly from the executor.
> > 
> > I think if you have developers running a small wrapper locally to deploy this new development stack you should run
> > that same wrapper in CI. This ensure the wrapper doesn't break.
> > 
> > > 
> > > # So how could we do this?
> > > The OpenStack Ansible project is made of many roles that are all
> > > composable, therefore, you can think of it as a combination of both
> > > Puppet-OpenStack and TripleO (back then).  Puppet-OpenStack contained
> > > the base modules (i.e. puppet-nova, etc) and TripleO was the
> > > integration of all of it in a distribution.  OSA is currently both,
> > > but it also includes both Ansible roles and playbooks.
> > > 
> > > In order to make sure we maintain as much of backwards compatibility
> > > as possible, we can simply run a small script which does a mapping of
> > > devstack => OSA variables to make sure that the service is shipped
> > > with all the necessary features as per local.conf.
> > > 
> > > So the new process could be:
> > > 
> > > 1) parse local.conf and generate Ansible variables files
> > > 2) install Ansible (if not running in gate)
> > > 3) run playbooks using variable generated in #1
> > > 
> > > The neat thing is after all of this, devstack just becomes a thin
> > > wrapper around Ansible roles.  I also think it brings a lot of hands
> > > together, involving both the QA team and OSA team together, which I
> > > believe that pooling our resources will greatly help in being able to
> > > get more done and avoiding duplicating our efforts.
> > > 
> > > # Conclusion
> > > This is a start of a very open ended discussion, I'm sure there is a
> > > lot of details involved here in the implementation that will surface,
> > > but I think it could be a good step overall in simplifying our CI and
> > > adding more coverage for real potential deployers.  It will help two
> > > teams unite together and have more resources for something (that
> > > essentially is somewhat of duplicated effort at the moment).
> > > 
> > > I will try to pick up sometime to POC a simple service being deployed
> > > by an OSA role instead of Bash, placement which seems like a very
> > > simple one and share that eventually.
> > > 
> > > Thoughts? :)
> > 
> > For me there are two major items to consider that haven't been brought up yet. The first is devstack's (lack of)
> > speed. Any replacement should be at least as quick as the current tooling because the current tooling is slow enough
> > already.
> > 
> > This is important. We would need to see benchmark comparisons between a devstack install and an OSA install. Shell
> > may be slow but Ansible is generally slower. That's fine in production when reliability is king, but we need fast
> > iteration for development.
> > 
> > I haven't looked under the covers of devstack for some time, but it previously installed all python deps in one
> > place, whereas OSA has virtualenvs for each service which could take a while to build. Perhaps this is configurable.
> > 
> > The other is logging. I spend a lot of time helping people to debug CI job runs and devstack has grown a fairly
> > effective set of logging that just about any time I have to help debug another deployment tool's CI jobs I miss
> > (because they tend to log only a tiny fraction of what devstack logs).
> > 
> > Clark
> 
>