Open Stack

On Tue, 2019-06-04 at 08:39 -0400, Doug Hellmann wrote:
> Sean Mooney <smooney at redhat.com> writes:
> 
> > On Mon, 2019-06-03 at 08:39 -0400, Mohammed Naser wrote:
> > > On Mon, Jun 3, 2019 at 8:27 AM Slawomir Kaplonski <skaplons at redhat.com> wrote:
> > > > 
> > > > Hi,
> > > > 
> > > > > On 1 Jun 2019, at 20:49, Mohammed Naser <mnaser at vexxhost.com> wrote:
> > > > > 
> > > > > On Sat, Jun 1, 2019 at 1:46 PM Slawomir Kaplonski <skaplons at redhat.com> wrote:
> > > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > I don’t know OSA at all so sorry if my question is dumb but in devstack we can easily write plugins, keep it
> > > > > > in
> > > > > > separate repo and such plugin can be easy used in devstack (e.g. in CI jobs it’s used a lot). Is something
> > > > > > similar possible with OSA or will it be needed to contribute always every change to OSA repository?
> > > > > 
> > > > > Not a dumb question at all.  So, we do have this concept of 'roles'
> > > > > which you _could_ kinda technically identify similar to plugins.
> > > > > However, I think one of the things that would maybe come out of this
> > > > > is the inability for projects to maintain their own plugins (because
> > > > > now you can host neutron/devstack/plugins and you maintain that repo
> > > > > yourself), under this structure, you would indeed have to make those
> > > > > changes to the OpenStack Ansible Neutron role
> > > > > 
> > > > > i.e.: https://opendev.org/openstack/openstack-ansible-os_neutron
> > > > > 
> > > > > However, I think from an OSA perspective, we would be more than happy
> > > > > to add project maintainers for specific projects to their appropriate
> > > > > roles.  It would make sense that there is someone from the Neutron
> > > > > team that could be a core on os_neutron from example.
> > > > 
> > > > Yes, that may work for official projects like Neutron. But what about everything else, like projects hosted now
> > > > in
> > > > opendev.org/x/ repositories? Devstack gives everyone easy way to integrate own plugin/driver/project with it and
> > > > install it together with everything else by simply adding one line (usually) in local.conf file.
> > > > I think that it may be a bit hard to OSA team to accept and review patches with new roles for every project or
> > > > driver which isn’t official OpenStack project.
> > > 
> > > You raise a really good concern.  Indeed, we might have to change the workflow
> > > from "write a plugin" to "write an Ansible role" to be able to test
> > > your project with
> > > DevStack at that page (or maintain both a "legacy" solution) with a new one.
> > 
> > the real probalem with that is who is going to port all of the
> > existing plugins.
> 
> Do all projects and all jobs have to be converted at once? Or ever?
> 
> How much complexity do those plugins actually contain? Would they be
> fairly straightforward to convert?
that depends. some jsut add support for indivigual projects.
others install infrastructure services like ceph or kubernetes which will be used by openstack
services. others download and compiles c projects form source like networking-ovs-dpdk.
the neutron devstack pluging also used to compiles ovs form source to work around some distro bugs
and networking-ovn i belive also can? do the same.
a devstack plugin allows all of the above to be done trivally.

> 
> Could we build a "devstack plugin wrapper" for OSA? Could we run OSA and
> then run devstack with just the plugin(s) needed for a given job?
that would likely be possible. im sure we could generate local.conf  form osa's inventories
and and run the plugsins after osa runs. devstack always runs it in tree code in each phase and
then runs the plugins in the order they are enabled in each phase

https://docs.openstack.org/devstack/latest/plugins.html


networking-ovs-dpdk for example replaces the _neutron_ovs_base_install_agent_packages function
https://github.com/openstack/networking-ovs-dpdk/blob/master/devstack/libs/ovs-dpdk#L11-L16
with a noop and then in the install pahse we install ovs-dpdk form souce.
_neutron_ovs_base_install_agent_packages just install kernel ovs but we replace it as
our patches to make it condtional in devstack were rejected.

its not nessiarily a patteren i encurage but if you have to you can replace any functionality
that devstack provides via a plugin although most usecase relly dont requrie that.

> 
> Is there enough appeal in the idea of replacing devstack with something
> closer to what is used for production deployments to drive us to find an
> iterative approach that doesn't require changing everything at one time?
> Or are we stuck with devstack forever?
> 
> > kolla-ansible has also tried to be a devstack replacement in the past via the introduction
> > of dev-mode which clones the git repo of the dev mode project locally and bind mounts them into the container.
> > the problem is it still breaks peoles plugins and workflow.
> > 
> > 
> > some devstack feature that osa would need to support in order to be a
> > replacement for me are.
> 
> You've made a good start on a requirements list for a devstack
> replacement. Perhaps a first step would be for some of the folks who
> support this idea to compile a more complete list of those requirements,
> and then we could analyze OSA to see how it might need to be changed or
> whether it makes sense to use OSA as the basis for a new toolset that
> takes on some of the "dev" features we might not want in a "production"
> deployment tool.
> 
> Here's another potential gap for whoever is going to make that list:
> devstack pre-populates the environment with some data for things like
> flavors and images. I don't imagine OSA does that or, if it does, that
> they are an exact match. How do we change those settings?
+1 yes this is somehting i forgot about
> 
> That leads to a good second step: Do the rest of the analysis to
> understand what it would take to set up a base job like we have for
> devstack, that produces a similar setup. Not necessarily identical, but
> similar enough to be able to run tempest. It seems likely that already
> exists in some form for testing OSA itself. Could a developer run that
> on a local system (clearly being able to build the test environment
> locally is a requirement for replacing devstack)?
> 
> After that, I would want to see answers to some of the questions about
> dealing with plugins that I posed above. 
> 
> And only then, I think, could I provide an answer to the question of
> whether we should make the change or not.
> 
> > 1 the ablity to install all openstack project form git if needed including gerrit reviews.
> > 
> > abiltiy to eailly specify gerrit reiews or commits for each project
> > 
> > # here i am declaring the os-vif should be installed from git not pypi
> > LIBS_FROM_GIT=os-vif
> > 
> > # and here i am specifying that gerrit should be used as the source and
> > # i am provide a gerrit/git refs branch for a specific un merged patch
> > OS_VIF_REPO=https://git.openstack.org/openstack/os-vif
> > OS_VIF_BRANCH=refs/changes/25/629025/9
> > 
> > # *_REPO can obvioulsy take anythign that is valid in a git clone command so
> > # i can use a local repo too
> > NEUTRON_REPO=file:///opt/repos/neutron
> > # and *_BRANCH as the name implices works with branches, tag commits* and gerrit ref brances.
> > NEUTRON_BRANCH=bug/1788009
> > 
> > 
> > the next thing that would be needed is a way to simply override any config value like this
> > 
> > [[post-config|/etc/nova/nova.conf]]
> > #[compute]
> > #live_migration_wait_for_vif_plug=True
> > [libvirt]
> > live_migration_uri = qemu+ssh://root@%s/system
> > #cpu_mode = host-passthrough
> > virt_type = kvm
> > cpu_mode = custom
> > cpu_model = kvm64
> > 
> > im sure that osa can do that but i really can just provide any path to any file if needed.
> > so no need to update a role or plugin to set values in files created
> > by plugins which is the next thing.
> 
> Does OSA need to support *every* configuration value? Or could it deploy
> a stack, and then rely on a separate tool to modify config values and
> restart a service? Clearly some values need to be there when the cloud
> first starts, but do they all?
i think to preserve the workflow yes we need to be able to override any config that is generated
by OSA. kolla ansible supports a relly nice config override mechanism where you can supply overrieds
are applied after it generates a template. even though i have used the generic functionality to change thing
like libvirt configs in the past i generally have only used it for the openstack services and for development
i think its very imporant to easibly configure different senarios without needing to lear the opinionated syntatic sugar
provided by the install and just set the config values directly especially when developing a new feature that adds a new
value.

> 
> > we enable plugins with a single line like this
> > 
> > enable_plugin networking-ovs-dpdk https://github.com/openstack/networking-ovs-dpdk master
> >  some bugs 
> > meaning there is no need to preinstall or clone the repo. in theory the plugin should install all its dependeices
> > and devstack will clone and execute the plugins based on the single
> > line above. plugins however can also
> 
> This makes me think it might be most appropriate to be considering a
> tool that replaces devstack by wrapping OSA, rather than *being*
> OSA. Maybe that's just an extra playbook that runs before OSA, or maybe
> it's a simpler bash script that does some setup before invoking OSA.
on that point i had considerd porting networking-ovs-dpdk to an ansible role and invoking from
the devstack plugin in the past but i have not had time to do that. part of what is
nice about devstack plugin model is you can write you plugin in any language you like provided
you have a plug.sh file as an entrypoint. i doublt we have devstack plugins today that just run ansibel or puppet
but it is totally valid to do so.
> 
> > read any varable defiend in the local.conf as it will be set in the environment which means i can easily share
> > an exact configuration with someone by shareing a local.conf.
> > 
> > 
> > im not against improving or replacing devstack but with the devstack ansible roles and the fact we use devstack
> > for all our testing in the gate it is actually has become one of the best openstack installer out there. we do
> > not recommend people run it in production but with the ansible automation of grenade and the move to systemd for
> > services there are less mainatined installers out there that devstack is proably a better foundation for a cloud
> > to build on. people should still not use it in production but i can see why some might.
> > 
> > > 
> > > > > 
> > > > > > Speaking about CI, e.g. in neutron we currently have jobs like neutron-functional or neutron-fullstack which
> > > > > > uses only some parts of devstack. That kind of jobs will probably have to be rewritten after such change. I
> > > > > > don’t know if neutron jobs are only which can be affected in that way but IMHO it’s something worth to keep
> > > > > > in
> > > > > > mind.
> > > > > 
> > > > > Indeed, with our current CI infrastructure with OSA, we have the
> > > > > ability to create these dynamic scenarios (which can actually be
> > > > > defined by a simple Zuul variable).
> > > > > 
> > > > > https://github.com/openstack/openstack-ansible/blob/master/zuul.d/playbooks/pre-gate-scenario.yml#L41-L46
> > > > > 
> > > > > We do some really neat introspection of the project name being tested
> > > > > in order to run specific scenarios.  Therefore, that is something that
> > > > > should be quite easy to accomplish simply by overriding a scenario
> > > > > name within Zuul.  It also is worth mentioning we now support full
> > > > > metal deploys for a while now, so not having to worry about containers
> > > > > is something to keep in mind as well (with simplifying the developer
> > > > > experience again).
> > > > > 
> > > > > > > On 1 Jun 2019, at 14:35, Mohammed Naser <mnaser at vexxhost.com> wrote:
> > > > > > > 
> > > > > > > Hi everyone,
> > > > > > > 
> > > > > > > This is something that I've discussed with a few people over time and
> > > > > > > I think I'd probably want to bring it up by now.  I'd like to propose
> > > > > > > and ask if it makes sense to perhaps replace devstack entirely with
> > > > > > > openstack-ansible.  I think I have quite a few compelling reasons to
> > > > > > > do this that I'd like to outline, as well as why I *feel* (and I could
> > > > > > > be biased here, so call me out!) that OSA is the best option in terms
> > > > > > > of a 'replacement'
> > > > > > > 
> > > > > > > # Why not another deployment project?
> > > > > > > I actually thought about this part too and considered this mainly for
> > > > > > > ease of use for a *developer*.
> > > > > > > 
> > > > > > > At this point, Puppet-OpenStack pretty much only deploys packages
> > > > > > > (which means that it has no build infrastructure, a developer can't
> > > > > > > just get $commit checked out and deployed).
> > > > > > > 
> > > > > > > TripleO uses Kolla containers AFAIK and those have to be pre-built
> > > > > > > beforehand, also, I feel they are much harder to use as a developer
> > > > > > > because if you want to make quick edits and restart services, you have
> > > > > > > to enter a container and make the edit there and somehow restart the
> > > > > > > service without the container going back to it's original state.
> > > > > > > Kolla-Ansible and the other combinations also suffer from the same
> > > > > > > "issue".
> > > > > > > 
> > > > > > > OpenStack Ansible is unique in the way that it pretty much just builds
> > > > > > > a virtualenv and installs packages inside of it.  The services are
> > > > > > > deployed as systemd units.  This is very much similar to the current
> > > > > > > state of devstack at the moment (minus the virtualenv part, afaik).
> > > > > > > It makes it pretty straight forward to go and edit code if you
> > > > > > > need/have to.  We also have support for Debian, CentOS, Ubuntu and
> > > > > > > SUSE.  This allows "devstack 2.0" to have far more coverage and make
> > > > > > > it much more easy to deploy on a wider variety of operating systems.
> > > > > > > It also has the ability to use commits checked out from Zuul so all
> > > > > > > the fancy Depends-On stuff we use works.
> > > > > > > 
> > > > > > > # Why do we care about this, I like my bash scripts!
> > > > > > > As someone who's been around for a *really* long time in OpenStack,
> > > > > > > I've seen a whole lot of really weird issues surface from the usage of
> > > > > > > DevStack to do CI gating.  For example, one of the recent things is
> > > > > > > the fact it relies on installing package-shipped noVNC, where as the
> > > > > > > 'master' noVNC has actually changed behavior a few months back and it
> > > > > > > is completely incompatible at this point (it's just a ticking thing
> > > > > > > until we realize we're entirely broken).
> > > > > > > 
> > > > > > > To this day, I still see people who want to POC something up with
> > > > > > > OpenStack or *ACTUALLY* try to run OpenStack with DevStack.  No matter
> > > > > > > how many warnings we'll put up, they'll always try to do it.  With
> > > > > > > this way, at least they'll have something that has the shape of an
> > > > > > > actual real deployment.  In addition, it would be *good* in the
> > > > > > > overall scheme of things for a deployment system to test against,
> > > > > > > because this would make sure things don't break in both ways.
> > > > > > > 
> > > > > > > Also: we run Zuul for our CI which supports Ansible natively, this can
> > > > > > > remove one layer of indirection (Zuul to run Bash) and have Zuul run
> > > > > > > the playbooks directly from the executor.
> > > > > > > 
> > > > > > > # So how could we do this?
> > > > > > > The OpenStack Ansible project is made of many roles that are all
> > > > > > > composable, therefore, you can think of it as a combination of both
> > > > > > > Puppet-OpenStack and TripleO (back then).  Puppet-OpenStack contained
> > > > > > > the base modules (i.e. puppet-nova, etc) and TripleO was the
> > > > > > > integration of all of it in a distribution.  OSA is currently both,
> > > > > > > but it also includes both Ansible roles and playbooks.
> > > > > > > 
> > > > > > > In order to make sure we maintain as much of backwards compatibility
> > > > > > > as possible, we can simply run a small script which does a mapping of
> > > > > > > devstack => OSA variables to make sure that the service is shipped
> > > > > > > with all the necessary features as per local.conf.
> > > > > > > 
> > > > > > > So the new process could be:
> > > > > > > 
> > > > > > > 1) parse local.conf and generate Ansible variables files
> > > > > > > 2) install Ansible (if not running in gate)
> > > > > > > 3) run playbooks using variable generated in #1
> > > > > > > 
> > > > > > > The neat thing is after all of this, devstack just becomes a thin
> > > > > > > wrapper around Ansible roles.  I also think it brings a lot of hands
> > > > > > > together, involving both the QA team and OSA team together, which I
> > > > > > > believe that pooling our resources will greatly help in being able to
> > > > > > > get more done and avoiding duplicating our efforts.
> > > > > > > 
> > > > > > > # Conclusion
> > > > > > > This is a start of a very open ended discussion, I'm sure there is a
> > > > > > > lot of details involved here in the implementation that will surface,
> > > > > > > but I think it could be a good step overall in simplifying our CI and
> > > > > > > adding more coverage for real potential deployers.  It will help two
> > > > > > > teams unite together and have more resources for something (that
> > > > > > > essentially is somewhat of duplicated effort at the moment).
> > > > > > > 
> > > > > > > I will try to pick up sometime to POC a simple service being deployed
> > > > > > > by an OSA role instead of Bash, placement which seems like a very
> > > > > > > simple one and share that eventually.
> > > > > > > 
> > > > > > > Thoughts? :)
> > > > > > > 
> > > > > > > --
> > > > > > > Mohammed Naser — vexxhost
> > > > > > > -----------------------------------------------------
> > > > > > > D. 514-316-8872
> > > > > > > D. 800-910-1726 ext. 200
> > > > > > > E. mnaser at vexxhost.com
> > > > > > > W. http://vexxhost.com
> > > > > > > 
> > > > > > 
> > > > > > —
> > > > > > Slawek Kaplonski
> > > > > > Senior software engineer
> > > > > > Red Hat
> > > > > > 
> > > > > 
> > > > > 
> > > > > --
> > > > > Mohammed Naser — vexxhost
> > > > > -----------------------------------------------------
> > > > > D. 514-316-8872
> > > > > D. 800-910-1726 ext. 200
> > > > > E. mnaser at vexxhost.com
> > > > > W. http://vexxhost.com
> > > > 
> > > > —
> > > > Slawek Kaplonski
> > > > Senior software engineer
> > > > Red Hat
> > > > 
> > > 
> > > 
> > 
> > 
> 
>