Open Stack

On Tue, Oct 13, 2015 at 6:13 AM, Vladimir Kuklin <vkuklin at mirantis.com>
wrote:

> Puppetmaster and Fuelers,
>
> Last week I mentioned that I would like to bring the theme of using native
> ruby OpenStack client and use it within the providers.
>
> Emilien told me that I had already been late and the decision was made
> that puppet-openstack decided to not work with Aviator based on [0]. I went
> through this thread and did not find any unresolvable issues with using
> Aviator in comparison with potential benefits it could have brought up.
>
> What I saw actually was like that:
>
> * Pros
>
> 1) It is a native ruby client
> 2) We can import it in puppet and use all the power of Ruby
> 3) We will not need to have a lot of forks/execs for puppet
> 4) You are relying on negotiated and structured output provided by API
> (JSON) instead of introducing workarounds for client output like [1]
>
> * Cons
>
> 1) Aviator is not actively supported
> 2) Aviator does not track all the upstream OpenStack features while native
> OpenStack client does support them
> 3) Ruby community is not really interested in OpenStack (this one is
> arguable, I think)
>
> * Proposed solution
>
> While I completely understand all the cons against using Aviator right
> now, I see that Pros above are essential enough to change our mind and
> invest our own resources into creating really good OpenStack binding in
> Ruby.
> Some are saying that there is not so big involvement of Ruby into
> OpenStack. But we are actually working with Puppet/Ruby and are invloved
> into community. So why should not we own this by ourselves and lead by
> example here?
>
> I understand that many of you do already have a lot of things on their
> plate and cannot or would not want to support things like additional
> library when native OpenStack client is working reasonably well for you.
> But if I propose the following scheme to get support of native Ruby client
> for OpenStack:
>
> 1) we (community) have these resources (speaking of the company I am
> working for, we at Mirantis have a set of guys who could be very interested
> in working on Ruby client for OpenStack)
> 2) we gradually improve Aviator code base up to the level that it
> eliminates issues that are mentioned in  'Cons' section
> 3) we introduce additional set of providers and allow users and operators
> to pick whichever they want
> 4) we leave OpenStackClient default one
>
> Would you support it and allow such code to be merged into upstream
> puppet-openstack modules?
>
>
> [0]
> https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
> [1]
> https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com <http://www.mirantis.ru/>
> www.mirantis.ru
> vkuklin at mirantis.com
>

The scale-tipping reason we went with python-openstackclient over the
Aviator library was that at the same time we were trying to switch, we were
also developing keystone v3 features and we could only get those features
from python-openstackclient.

For the first two pros you listed, I'm not convinced they're really pros.
Puppet types and providers are actually extremely well-suited to shelling
out to command-line clients. There are existing, documented puppet APIs to
do it and we get automatic debug output with it. Nearly every existing type
and provider does this. It is not well-suited to call out to other
non-standard ruby libraries because they need to be added as a dependency
somehow, and doing this is not well-established in puppet. There are
basically two options to do this:

 1) Add a gem as a package resource and make sure the package resource is
called before any of the openstack resources. I could see this working as
an opt-in thing, but not as a default, for the same reason we don't require
our users to install pip libraries - there is less security guarantees from
pypi and rubygems than from distro packages, plus corporate infrastructure
may not allow pulling packages from these types of sources. (I don't see
this policy documented anywhere, this was just something that's been
instilled in me since I started working on this team. If we want to revisit
it, formalize it, or drop it, we can start a separate thread for that.)

2) Vendor the gem as a module. This was how we tried it before, and you can
see aimonb/aviator <https://github.com/aimonb/puppet_aviator> for this. The
problems I see with this are that we have to keep the module in sync with
the gem, and there is no way to keep proper attribution in the module for
work that was really done in the gem.

I am not convinced that the fork/execs are really causing that much
performance issues, though I've heard from some people that
python-openstackclient itself is sort of slow. The puppet agent itself has
a lot of overhead that causes it to be slow, this is why puppetlabs is
planning to rewrite it in C++.

For pro #4, the example you gave isn't really valid because the swift
module isn't using openstackclient, and openstackclient has the ability to
output data in CSV or JSON. A valid and related complaint would be that
openstackclient sometimes spits out ugly irrelevant warnings that we have
to parse around, since puppet is terrible at splitting stdout from stderr.

I think a major issue we had the first time we tried to do this was that we
started doing work to rewrite these things without actually knowing if it
would be better. Instead of using that as a precedent for doing the same
thing again, I would like to take that experience and do this right by
actually gathering data: what is wrong with how we're doing it now, what
numbers and metrics show that things are bad? Can we create a
proof-of-concept that demonstrates that we can really improve these numbers?

If we decide that using a ruby library over the CLI is really a measurably
better approach, I would recommend we get in contact with Mark Maglana, the
author of Aviator, and get the library moved back over to openstack rather
than GitHub, and set up a core reviewer team with multiple members from
multiiple companies. We should also work with the OpenStack-SDK team to see
if we can get this moved under their project or if they have suggestions
for us.

Colleen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151013/ad14d3a1/attachment.html>