On Mon, Jun 3, 2019 at 8:02 AM Jim Rollenhagen <jim@jimrollenhagen.com> wrote:
I don't think I have enough coffee in me to fully digest this, but wanted to point out a couple of things. FWIW, this is something I've thought we should do for a while now.
On Sat, Jun 1, 2019 at 8:43 AM Mohammed Naser <mnaser@vexxhost.com> wrote:
Hi everyone,
This is something that I've discussed with a few people over time and I think I'd probably want to bring it up by now. I'd like to propose and ask if it makes sense to perhaps replace devstack entirely with openstack-ansible. I think I have quite a few compelling reasons to do this that I'd like to outline, as well as why I *feel* (and I could be biased here, so call me out!) that OSA is the best option in terms of a 'replacement'
# Why not another deployment project? I actually thought about this part too and considered this mainly for ease of use for a *developer*.
At this point, Puppet-OpenStack pretty much only deploys packages (which means that it has no build infrastructure, a developer can't just get $commit checked out and deployed).
TripleO uses Kolla containers AFAIK and those have to be pre-built beforehand, also, I feel they are much harder to use as a developer because if you want to make quick edits and restart services, you have to enter a container and make the edit there and somehow restart the service without the container going back to it's original state. Kolla-Ansible and the other combinations also suffer from the same "issue".
FWIW, kolla-ansible (and maybe tripleo?) has a "development" mode which mounts the code as a volume, so you can make edits and just run "docker restart $service". Though systemd does make that a bit nicer due to globs (e.g. systemctl restart nova-*).
That said, I do agree moving to something where systemd is running the services would make for a smoother transition for developers.
I didn't know about this (and this wasn't around for the time that I was trying and experimenting with Kolla). This does seem like a possible solution if we're okay with adding the Docker dependency into DevStack and the workflow changing from restarting services to restarting containers.
OpenStack Ansible is unique in the way that it pretty much just builds a virtualenv and installs packages inside of it. The services are deployed as systemd units. This is very much similar to the current state of devstack at the moment (minus the virtualenv part, afaik). It makes it pretty straight forward to go and edit code if you need/have to. We also have support for Debian, CentOS, Ubuntu and SUSE. This allows "devstack 2.0" to have far more coverage and make it much more easy to deploy on a wider variety of operating systems. It also has the ability to use commits checked out from Zuul so all the fancy Depends-On stuff we use works.
# Why do we care about this, I like my bash scripts! As someone who's been around for a *really* long time in OpenStack, I've seen a whole lot of really weird issues surface from the usage of DevStack to do CI gating. For example, one of the recent things is the fact it relies on installing package-shipped noVNC, where as the 'master' noVNC has actually changed behavior a few months back and it is completely incompatible at this point (it's just a ticking thing until we realize we're entirely broken).
To this day, I still see people who want to POC something up with OpenStack or *ACTUALLY* try to run OpenStack with DevStack. No matter how many warnings we'll put up, they'll always try to do it. With this way, at least they'll have something that has the shape of an actual real deployment. In addition, it would be *good* in the overall scheme of things for a deployment system to test against, because this would make sure things don't break in both ways.
++
Also: we run Zuul for our CI which supports Ansible natively, this can remove one layer of indirection (Zuul to run Bash) and have Zuul run the playbooks directly from the executor.
# So how could we do this? The OpenStack Ansible project is made of many roles that are all composable, therefore, you can think of it as a combination of both Puppet-OpenStack and TripleO (back then). Puppet-OpenStack contained the base modules (i.e. puppet-nova, etc) and TripleO was the integration of all of it in a distribution. OSA is currently both, but it also includes both Ansible roles and playbooks.
In order to make sure we maintain as much of backwards compatibility as possible, we can simply run a small script which does a mapping of devstack => OSA variables to make sure that the service is shipped with all the necessary features as per local.conf.
++
So the new process could be:
1) parse local.conf and generate Ansible variables files 2) install Ansible (if not running in gate) 3) run playbooks using variable generated in #1
The neat thing is after all of this, devstack just becomes a thin wrapper around Ansible roles. I also think it brings a lot of hands together, involving both the QA team and OSA team together, which I believe that pooling our resources will greatly help in being able to get more done and avoiding duplicating our efforts.
# Conclusion This is a start of a very open ended discussion, I'm sure there is a lot of details involved here in the implementation that will surface, but I think it could be a good step overall in simplifying our CI and adding more coverage for real potential deployers. It will help two teams unite together and have more resources for something (that essentially is somewhat of duplicated effort at the moment).
I will try to pick up sometime to POC a simple service being deployed by an OSA role instead of Bash, placement which seems like a very simple one and share that eventually.
Thoughts? :)
The reason this hasn't been pushed on in the past is to avoid the perception that the TC or QA team is choosing a "winner" in the deployment space. I don't think that's a good reason not to do something like this (especially with the drop in contributors since I've had that discussion). However, we do need to message this carefully at a minimum.
Right. I think that's because in OpenStack-Ansible world, we have two things - OSA roles: nothing but basic roles to deploy OpenStack services, with external consumers - Integrated: contains all the playbooks In a way, our roles is "Puppet OpenStack" and our integrated repo is "TripleO", back when TripleO deployed via Puppet anyways... I have to be honest, I wish that our roles lived under a different name so we can collaborate all on them (because an Ansible role to deploy something generically is needed regardless). We've actually done a lot of work with the TripleO team and they are consuming one of our roles (os_tempest) to do all their tempest testing, we gate TripleO and they gate us for the role.
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. http://vexxhost.com
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. http://vexxhost.com