Open Stack

On 6/4/19 7:39 AM, Doug Hellmann wrote:
> Sean Mooney <smooney at redhat.com> writes:
> 
>> On Mon, 2019-06-03 at 08:39 -0400, Mohammed Naser wrote:
>>> On Mon, Jun 3, 2019 at 8:27 AM Slawomir Kaplonski <skaplons at redhat.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>>> On 1 Jun 2019, at 20:49, Mohammed Naser <mnaser at vexxhost.com> wrote:
>>>>>
>>>>> On Sat, Jun 1, 2019 at 1:46 PM Slawomir Kaplonski <skaplons at redhat.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I don’t know OSA at all so sorry if my question is dumb but in devstack we can easily write plugins, keep it in
>>>>>> separate repo and such plugin can be easy used in devstack (e.g. in CI jobs it’s used a lot). Is something
>>>>>> similar possible with OSA or will it be needed to contribute always every change to OSA repository?
>>>>>
>>>>> Not a dumb question at all.  So, we do have this concept of 'roles'
>>>>> which you _could_ kinda technically identify similar to plugins.
>>>>> However, I think one of the things that would maybe come out of this
>>>>> is the inability for projects to maintain their own plugins (because
>>>>> now you can host neutron/devstack/plugins and you maintain that repo
>>>>> yourself), under this structure, you would indeed have to make those
>>>>> changes to the OpenStack Ansible Neutron role
>>>>>
>>>>> i.e.: https://opendev.org/openstack/openstack-ansible-os_neutron
>>>>>
>>>>> However, I think from an OSA perspective, we would be more than happy
>>>>> to add project maintainers for specific projects to their appropriate
>>>>> roles.  It would make sense that there is someone from the Neutron
>>>>> team that could be a core on os_neutron from example.
>>>>
>>>> Yes, that may work for official projects like Neutron. But what about everything else, like projects hosted now in
>>>> opendev.org/x/ repositories? Devstack gives everyone easy way to integrate own plugin/driver/project with it and
>>>> install it together with everything else by simply adding one line (usually) in local.conf file.
>>>> I think that it may be a bit hard to OSA team to accept and review patches with new roles for every project or
>>>> driver which isn’t official OpenStack project.
>>>
>>> You raise a really good concern.  Indeed, we might have to change the workflow
>>> from "write a plugin" to "write an Ansible role" to be able to test
>>> your project with
>>> DevStack at that page (or maintain both a "legacy" solution) with a new one.
>> the real probalem with that is who is going to port all of the
>> existing plugins.
> 
> Do all projects and all jobs have to be converted at once? Or ever?

Perhaps not all at once, but I would say they all need to be converted 
eventually or we end up in the situation Dean mentioned where we have to 
maintain two different deployment systems. I would argue that's much 
worse than just continuing with devstack as-is. On the other hand, 
practically speaking I don't think we can probably do them all at once, 
unless there are a lot fewer devstack plugins in the wild than I think 
there are (which is possible). Also, I suspect there may be downstream 
plugins running in third-party ci that need to be considered.

That said, while I expect this would be _extremely_ painful in the short 
to medium term, I'm also a big proponent of making the thing developers 
care about the same as the thing users care about. However, if we go 
down this path I think we need sufficient buy in from a diverse enough 
group of contributors that losing one group (see OSIC) doesn't leave us 
with a half-finished migration. That would be a disaster IMHO.

> 
> How much complexity do those plugins actually contain? Would they be
> fairly straightforward to convert?
> 
> Could we build a "devstack plugin wrapper" for OSA? Could we run OSA and
> then run devstack with just the plugin(s) needed for a given job?
> 
> Is there enough appeal in the idea of replacing devstack with something
> closer to what is used for production deployments to drive us to find an
> iterative approach that doesn't require changing everything at one time?
> Or are we stuck with devstack forever?
> 
>> kolla-ansible has also tried to be a devstack replacement in the past via the introduction
>> of dev-mode which clones the git repo of the dev mode project locally and bind mounts them into the container.
>> the problem is it still breaks peoles plugins and workflow.
>>
>>
>> some devstack feature that osa would need to support in order to be a
>> replacement for me are.
> 
> You've made a good start on a requirements list for a devstack
> replacement. Perhaps a first step would be for some of the folks who
> support this idea to compile a more complete list of those requirements,
> and then we could analyze OSA to see how it might need to be changed or
> whether it makes sense to use OSA as the basis for a new toolset that
> takes on some of the "dev" features we might not want in a "production"
> deployment tool.
> 
> Here's another potential gap for whoever is going to make that list:
> devstack pre-populates the environment with some data for things like
> flavors and images. I don't imagine OSA does that or, if it does, that
> they are an exact match. How do we change those settings?
> 
> That leads to a good second step: Do the rest of the analysis to
> understand what it would take to set up a base job like we have for
> devstack, that produces a similar setup. Not necessarily identical, but
> similar enough to be able to run tempest. It seems likely that already
> exists in some form for testing OSA itself. Could a developer run that
> on a local system (clearly being able to build the test environment
> locally is a requirement for replacing devstack)?
> 
> After that, I would want to see answers to some of the questions about
> dealing with plugins that I posed above.
> 
> And only then, I think, could I provide an answer to the question of
> whether we should make the change or not.
> 
>> 1 the ablity to install all openstack project form git if needed including gerrit reviews.
>>
>> abiltiy to eailly specify gerrit reiews or commits for each project
>>
>> # here i am declaring the os-vif should be installed from git not pypi
>> LIBS_FROM_GIT=os-vif
>>
>> # and here i am specifying that gerrit should be used as the source and
>> # i am provide a gerrit/git refs branch for a specific un merged patch
>> OS_VIF_REPO=https://git.openstack.org/openstack/os-vif
>> OS_VIF_BRANCH=refs/changes/25/629025/9
>>
>> # *_REPO can obvioulsy take anythign that is valid in a git clone command so
>> # i can use a local repo too
>> NEUTRON_REPO=file:///opt/repos/neutron
>> # and *_BRANCH as the name implices works with branches, tag commits* and gerrit ref brances.
>> NEUTRON_BRANCH=bug/1788009
>>
>>
>> the next thing that would be needed is a way to simply override any config value like this
>>
>> [[post-config|/etc/nova/nova.conf]]
>> #[compute]
>> #live_migration_wait_for_vif_plug=True
>> [libvirt]
>> live_migration_uri = qemu+ssh://root@%s/system
>> #cpu_mode = host-passthrough
>> virt_type = kvm
>> cpu_mode = custom
>> cpu_model = kvm64
>>
>> im sure that osa can do that but i really can just provide any path to any file if needed.
>> so no need to update a role or plugin to set values in files created
>> by plugins which is the next thing.
> 
> Does OSA need to support *every* configuration value? Or could it deploy
> a stack, and then rely on a separate tool to modify config values and
> restart a service? Clearly some values need to be there when the cloud
> first starts, but do they all?
> 
>> we enable plugins with a single line like this
>>
>> enable_plugin networking-ovs-dpdk https://github.com/openstack/networking-ovs-dpdk master
>>
>> meaning there is no need to preinstall or clone the repo. in theory the plugin should install all its dependeices
>> and devstack will clone and execute the plugins based on the single
>> line above. plugins however can also
> 
> This makes me think it might be most appropriate to be considering a
> tool that replaces devstack by wrapping OSA, rather than *being*
> OSA. Maybe that's just an extra playbook that runs before OSA, or maybe
> it's a simpler bash script that does some setup before invoking OSA.
> 
>> read any varable defiend in the local.conf as it will be set in the environment which means i can easily share
>> an exact configuration with someone by shareing a local.conf.
>>
>>
>> im not against improving or replacing devstack but with the devstack ansible roles and the fact we use devstack
>> for all our testing in the gate it is actually has become one of the best openstack installer out there. we do
>> not recommend people run it in production but with the ansible automation of grenade and the move to systemd for
>> services there are less mainatined installers out there that devstack is proably a better foundation for a cloud
>> to build on. people should still not use it in production but i can see why some might.
>>
>>>
>>>>>
>>>>>> Speaking about CI, e.g. in neutron we currently have jobs like neutron-functional or neutron-fullstack which
>>>>>> uses only some parts of devstack. That kind of jobs will probably have to be rewritten after such change. I
>>>>>> don’t know if neutron jobs are only which can be affected in that way but IMHO it’s something worth to keep in
>>>>>> mind.
>>>>>
>>>>> Indeed, with our current CI infrastructure with OSA, we have the
>>>>> ability to create these dynamic scenarios (which can actually be
>>>>> defined by a simple Zuul variable).
>>>>>
>>>>> https://github.com/openstack/openstack-ansible/blob/master/zuul.d/playbooks/pre-gate-scenario.yml#L41-L46
>>>>>
>>>>> We do some really neat introspection of the project name being tested
>>>>> in order to run specific scenarios.  Therefore, that is something that
>>>>> should be quite easy to accomplish simply by overriding a scenario
>>>>> name within Zuul.  It also is worth mentioning we now support full
>>>>> metal deploys for a while now, so not having to worry about containers
>>>>> is something to keep in mind as well (with simplifying the developer
>>>>> experience again).
>>>>>
>>>>>>> On 1 Jun 2019, at 14:35, Mohammed Naser <mnaser at vexxhost.com> wrote:
>>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> This is something that I've discussed with a few people over time and
>>>>>>> I think I'd probably want to bring it up by now.  I'd like to propose
>>>>>>> and ask if it makes sense to perhaps replace devstack entirely with
>>>>>>> openstack-ansible.  I think I have quite a few compelling reasons to
>>>>>>> do this that I'd like to outline, as well as why I *feel* (and I could
>>>>>>> be biased here, so call me out!) that OSA is the best option in terms
>>>>>>> of a 'replacement'
>>>>>>>
>>>>>>> # Why not another deployment project?
>>>>>>> I actually thought about this part too and considered this mainly for
>>>>>>> ease of use for a *developer*.
>>>>>>>
>>>>>>> At this point, Puppet-OpenStack pretty much only deploys packages
>>>>>>> (which means that it has no build infrastructure, a developer can't
>>>>>>> just get $commit checked out and deployed).
>>>>>>>
>>>>>>> TripleO uses Kolla containers AFAIK and those have to be pre-built
>>>>>>> beforehand, also, I feel they are much harder to use as a developer
>>>>>>> because if you want to make quick edits and restart services, you have
>>>>>>> to enter a container and make the edit there and somehow restart the
>>>>>>> service without the container going back to it's original state.
>>>>>>> Kolla-Ansible and the other combinations also suffer from the same
>>>>>>> "issue".
>>>>>>>
>>>>>>> OpenStack Ansible is unique in the way that it pretty much just builds
>>>>>>> a virtualenv and installs packages inside of it.  The services are
>>>>>>> deployed as systemd units.  This is very much similar to the current
>>>>>>> state of devstack at the moment (minus the virtualenv part, afaik).
>>>>>>> It makes it pretty straight forward to go and edit code if you
>>>>>>> need/have to.  We also have support for Debian, CentOS, Ubuntu and
>>>>>>> SUSE.  This allows "devstack 2.0" to have far more coverage and make
>>>>>>> it much more easy to deploy on a wider variety of operating systems.
>>>>>>> It also has the ability to use commits checked out from Zuul so all
>>>>>>> the fancy Depends-On stuff we use works.
>>>>>>>
>>>>>>> # Why do we care about this, I like my bash scripts!
>>>>>>> As someone who's been around for a *really* long time in OpenStack,
>>>>>>> I've seen a whole lot of really weird issues surface from the usage of
>>>>>>> DevStack to do CI gating.  For example, one of the recent things is
>>>>>>> the fact it relies on installing package-shipped noVNC, where as the
>>>>>>> 'master' noVNC has actually changed behavior a few months back and it
>>>>>>> is completely incompatible at this point (it's just a ticking thing
>>>>>>> until we realize we're entirely broken).
>>>>>>>
>>>>>>> To this day, I still see people who want to POC something up with
>>>>>>> OpenStack or *ACTUALLY* try to run OpenStack with DevStack.  No matter
>>>>>>> how many warnings we'll put up, they'll always try to do it.  With
>>>>>>> this way, at least they'll have something that has the shape of an
>>>>>>> actual real deployment.  In addition, it would be *good* in the
>>>>>>> overall scheme of things for a deployment system to test against,
>>>>>>> because this would make sure things don't break in both ways.
>>>>>>>
>>>>>>> Also: we run Zuul for our CI which supports Ansible natively, this can
>>>>>>> remove one layer of indirection (Zuul to run Bash) and have Zuul run
>>>>>>> the playbooks directly from the executor.
>>>>>>>
>>>>>>> # So how could we do this?
>>>>>>> The OpenStack Ansible project is made of many roles that are all
>>>>>>> composable, therefore, you can think of it as a combination of both
>>>>>>> Puppet-OpenStack and TripleO (back then).  Puppet-OpenStack contained
>>>>>>> the base modules (i.e. puppet-nova, etc) and TripleO was the
>>>>>>> integration of all of it in a distribution.  OSA is currently both,
>>>>>>> but it also includes both Ansible roles and playbooks.
>>>>>>>
>>>>>>> In order to make sure we maintain as much of backwards compatibility
>>>>>>> as possible, we can simply run a small script which does a mapping of
>>>>>>> devstack => OSA variables to make sure that the service is shipped
>>>>>>> with all the necessary features as per local.conf.
>>>>>>>
>>>>>>> So the new process could be:
>>>>>>>
>>>>>>> 1) parse local.conf and generate Ansible variables files
>>>>>>> 2) install Ansible (if not running in gate)
>>>>>>> 3) run playbooks using variable generated in #1
>>>>>>>
>>>>>>> The neat thing is after all of this, devstack just becomes a thin
>>>>>>> wrapper around Ansible roles.  I also think it brings a lot of hands
>>>>>>> together, involving both the QA team and OSA team together, which I
>>>>>>> believe that pooling our resources will greatly help in being able to
>>>>>>> get more done and avoiding duplicating our efforts.
>>>>>>>
>>>>>>> # Conclusion
>>>>>>> This is a start of a very open ended discussion, I'm sure there is a
>>>>>>> lot of details involved here in the implementation that will surface,
>>>>>>> but I think it could be a good step overall in simplifying our CI and
>>>>>>> adding more coverage for real potential deployers.  It will help two
>>>>>>> teams unite together and have more resources for something (that
>>>>>>> essentially is somewhat of duplicated effort at the moment).
>>>>>>>
>>>>>>> I will try to pick up sometime to POC a simple service being deployed
>>>>>>> by an OSA role instead of Bash, placement which seems like a very
>>>>>>> simple one and share that eventually.
>>>>>>>
>>>>>>> Thoughts? :)
>>>>>>>
>>>>>>> --
>>>>>>> Mohammed Naser — vexxhost
>>>>>>> -----------------------------------------------------
>>>>>>> D. 514-316-8872
>>>>>>> D. 800-910-1726 ext. 200
>>>>>>> E. mnaser at vexxhost.com
>>>>>>> W. http://vexxhost.com
>>>>>>>
>>>>>>
>>>>>> —
>>>>>> Slawek Kaplonski
>>>>>> Senior software engineer
>>>>>> Red Hat
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mohammed Naser — vexxhost
>>>>> -----------------------------------------------------
>>>>> D. 514-316-8872
>>>>> D. 800-910-1726 ext. 200
>>>>> E. mnaser at vexxhost.com
>>>>> W. http://vexxhost.com
>>>>
>>>> —
>>>> Slawek Kaplonski
>>>> Senior software engineer
>>>> Red Hat
>>>>
>>>
>>>
>>
>>
>