[qa][openstack-ansible] redefining devstack

Mark Goddard mark at stackhpc.com
Mon Jun 3 18:28:02 UTC 2019

On Mon, 3 Jun 2019, 15:59 Clark Boylan, <cboylan at sapwetik.org> wrote:

> On Sat, Jun 1, 2019, at 5:36 AM, Mohammed Naser wrote:
> > Hi everyone,
> >
> > This is something that I've discussed with a few people over time and
> > I think I'd probably want to bring it up by now.  I'd like to propose
> > and ask if it makes sense to perhaps replace devstack entirely with
> > openstack-ansible.  I think I have quite a few compelling reasons to
> > do this that I'd like to outline, as well as why I *feel* (and I could
> > be biased here, so call me out!) that OSA is the best option in terms
> > of a 'replacement'
> >
> > # Why not another deployment project?
> > I actually thought about this part too and considered this mainly for
> > ease of use for a *developer*.
> >
> > At this point, Puppet-OpenStack pretty much only deploys packages
> > (which means that it has no build infrastructure, a developer can't
> > just get $commit checked out and deployed).
> >
> > TripleO uses Kolla containers AFAIK and those have to be pre-built
> > beforehand, also, I feel they are much harder to use as a developer
> > because if you want to make quick edits and restart services, you have
> > to enter a container and make the edit there and somehow restart the
> > service without the container going back to it's original state.
> > Kolla-Ansible and the other combinations also suffer from the same
> > "issue".
> >
> > OpenStack Ansible is unique in the way that it pretty much just builds
> > a virtualenv and installs packages inside of it.  The services are
> > deployed as systemd units.  This is very much similar to the current
> > state of devstack at the moment (minus the virtualenv part, afaik).
> > It makes it pretty straight forward to go and edit code if you
> > need/have to.  We also have support for Debian, CentOS, Ubuntu and
> > SUSE.  This allows "devstack 2.0" to have far more coverage and make
> > it much more easy to deploy on a wider variety of operating systems.
> > It also has the ability to use commits checked out from Zuul so all
> > the fancy Depends-On stuff we use works.
> >
> > # Why do we care about this, I like my bash scripts!
> > As someone who's been around for a *really* long time in OpenStack,
> > I've seen a whole lot of really weird issues surface from the usage of
> > DevStack to do CI gating.  For example, one of the recent things is
> > the fact it relies on installing package-shipped noVNC, where as the
> > 'master' noVNC has actually changed behavior a few months back and it
> > is completely incompatible at this point (it's just a ticking thing
> > until we realize we're entirely broken).
> I'm not sure this is a great example case. We consume prebuilt software
> for many of our dependencies. Everything from the kernel to the database to
> rabbitmq to ovs (and so on) are consumed as prebuilt packages from our
> distros. In many cases this is desirable to ensure that our software work
> with the other software out there in the wild that people will be deploying
> with.
> >
> > To this day, I still see people who want to POC something up with
> > OpenStack or *ACTUALLY* try to run OpenStack with DevStack.  No matter
> > how many warnings we'll put up, they'll always try to do it.  With
> > this way, at least they'll have something that has the shape of an
> > actual real deployment.  In addition, it would be *good* in the
> > overall scheme of things for a deployment system to test against,
> > because this would make sure things don't break in both ways.
> >
> > Also: we run Zuul for our CI which supports Ansible natively, this can
> > remove one layer of indirection (Zuul to run Bash) and have Zuul run
> > the playbooks directly from the executor.
> I think if you have developers running a small wrapper locally to deploy
> this new development stack you should run that same wrapper in CI. This
> ensure the wrapper doesn't break.
> >
> > # So how could we do this?
> > The OpenStack Ansible project is made of many roles that are all
> > composable, therefore, you can think of it as a combination of both
> > Puppet-OpenStack and TripleO (back then).  Puppet-OpenStack contained
> > the base modules (i.e. puppet-nova, etc) and TripleO was the
> > integration of all of it in a distribution.  OSA is currently both,
> > but it also includes both Ansible roles and playbooks.
> >
> > In order to make sure we maintain as much of backwards compatibility
> > as possible, we can simply run a small script which does a mapping of
> > devstack => OSA variables to make sure that the service is shipped
> > with all the necessary features as per local.conf.
> >
> > So the new process could be:
> >
> > 1) parse local.conf and generate Ansible variables files
> > 2) install Ansible (if not running in gate)
> > 3) run playbooks using variable generated in #1
> >
> > The neat thing is after all of this, devstack just becomes a thin
> > wrapper around Ansible roles.  I also think it brings a lot of hands
> > together, involving both the QA team and OSA team together, which I
> > believe that pooling our resources will greatly help in being able to
> > get more done and avoiding duplicating our efforts.
> >
> > # Conclusion
> > This is a start of a very open ended discussion, I'm sure there is a
> > lot of details involved here in the implementation that will surface,
> > but I think it could be a good step overall in simplifying our CI and
> > adding more coverage for real potential deployers.  It will help two
> > teams unite together and have more resources for something (that
> > essentially is somewhat of duplicated effort at the moment).
> >
> > I will try to pick up sometime to POC a simple service being deployed
> > by an OSA role instead of Bash, placement which seems like a very
> > simple one and share that eventually.
> >
> > Thoughts? :)
> For me there are two major items to consider that haven't been brought up
> yet. The first is devstack's (lack of) speed. Any replacement should be at
> least as quick as the current tooling because the current tooling is slow
> enough already.

This is important. We would need to see benchmark comparisons between a
devstack install and an OSA install. Shell may be slow but Ansible is
generally slower. That's fine in production when reliability is king, but
we need fast iteration for development.

I haven't looked under the covers of devstack for some time, but it
previously installed all python deps in one place, whereas OSA has
virtualenvs for each service which could take a while to build. Perhaps
this is configurable.

The other is logging. I spend a lot of time helping people to debug CI job
> runs and devstack has grown a fairly effective set of logging that just
> about any time I have to help debug another deployment tool's CI jobs I
> miss (because they tend to log only a tiny fraction of what devstack logs).
> Clark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190603/2ff7c641/attachment-0001.html>

More information about the openstack-discuss mailing list