[openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers
Rich Megginson
rmeggins at redhat.com
Fri Oct 23 13:57:18 UTC 2015
On 10/22/2015 11:09 PM, Gilles Dubreuil wrote:
>
> On 21/10/15 00:56, Sofer Athlan-Guyot wrote:
>> Gilles Dubreuil <gilles at redhat.com> writes:
>>
>>> On 14/10/15 17:15, Gilles Dubreuil wrote:
>>>>
>>>> On 14/10/15 10:36, Colleen Murphy wrote:
>>>>>
>>>>> On Tue, Oct 13, 2015 at 6:13 AM, Vladimir Kuklin <vkuklin at mirantis.com
>>>>> <mailto:vkuklin at mirantis.com>> wrote:
>>>>>
>>>>> Puppetmaster and Fuelers,
>>>>>
>>>>> Last week I mentioned that I would like to bring the theme of using
>>>>> native ruby OpenStack client and use it within the providers.
>>>>>
>>>>> Emilien told me that I had already been late and the decision was
>>>>> made that puppet-openstack decided to not work with Aviator based on
>>>>> [0]. I went through this thread and did not find any unresolvable
>>>>> issues with using Aviator in comparison with potential benefits it
>>>>> could have brought up.
>>>>>
>>>>> What I saw actually was like that:
>>>>>
>>>>> * Pros
>>>>>
>>>>> 1) It is a native ruby client
>>>>> 2) We can import it in puppet and use all the power of Ruby
>>>>> 3) We will not need to have a lot of forks/execs for puppet
>>>>> 4) You are relying on negotiated and structured output provided by
>>>>> API (JSON) instead of introducing workarounds for client output like [1]
>>>>>
>>>>> * Cons
>>>>>
>>>>> 1) Aviator is not actively supported
>>>>> 2) Aviator does not track all the upstream OpenStack features while
>>>>> native OpenStack client does support them
>>>>> 3) Ruby community is not really interested in OpenStack (this one is
>>>>> arguable, I think)
>>>>>
>>>>> * Proposed solution
>>>>>
>>>>> While I completely understand all the cons against using Aviator
>>>>> right now, I see that Pros above are essential enough to change our
>>>>> mind and invest our own resources into creating really good
>>>>> OpenStack binding in Ruby.
>>>>> Some are saying that there is not so big involvement of Ruby into
>>>>> OpenStack. But we are actually working with Puppet/Ruby and are
>>>>> invloved into community. So why should not we own this by ourselves
>>>>> and lead by example here?
>>>>>
>>>>> I understand that many of you do already have a lot of things on
>>>>> their plate and cannot or would not want to support things like
>>>>> additional library when native OpenStack client is working
>>>>> reasonably well for you. But if I propose the following scheme to
>>>>> get support of native Ruby client for OpenStack:
>>>>>
>>>>> 1) we (community) have these resources (speaking of the company I am
>>>>> working for, we at Mirantis have a set of guys who could be very
>>>>> interested in working on Ruby client for OpenStack)
>>>>> 2) we gradually improve Aviator code base up to the level that it
>>>>> eliminates issues that are mentioned in 'Cons' section
>>>>> 3) we introduce additional set of providers and allow users and
>>>>> operators to pick whichever they want
>>>>> 4) we leave OpenStackClient default one
>>>>>
>>>>> Would you support it and allow such code to be merged into upstream
>>>>> puppet-openstack modules?
>>>>>
>>>>>
>>>>> [0] https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
>>>>> [1] https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
>>>>> --
>>>>> Yours Faithfully,
>>>>> Vladimir Kuklin,
>>>>> Fuel Library Tech Lead,
>>>>> Mirantis, Inc.
>>>>> +7 (495) 640-49-04 <tel:%2B7%20%28495%29%20640-49-04>
>>>>> +7 (926) 702-39-68 <tel:%2B7%20%28926%29%20702-39-68>
>>>>> Skype kuklinvv
>>>>> 35bk3, Vorontsovskaya Str.
>>>>> Moscow, Russia,
>>>>> www.mirantis.com <http://www.mirantis.ru/>
>>>>> www.mirantis.ru <http://www.mirantis.ru/>
>>>>> vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>
>>>>>
>>>>>
>>>>> The scale-tipping reason we went with python-openstackclient over the
>>>>> Aviator library was that at the same time we were trying to switch, we
>>>>> were also developing keystone v3 features and we could only get those
>>>>> features from python-openstackclient.
>>>>>
>>>>> For the first two pros you listed, I'm not convinced they're really
>>>>> pros. Puppet types and providers are actually extremely well-suited to
>>>>> shelling out to command-line clients. There are existing, documented
>>>>> puppet APIs to do it and we get automatic debug output with it. Nearly
>>>>> every existing type and provider does this. It is not well-suited to
>>>>> call out to other non-standard ruby libraries because they need to be
>>>>> added as a dependency somehow, and doing this is not well-established in
>>>>> puppet. There are basically two options to do this:
>>>>>
>>>>> 1) Add a gem as a package resource and make sure the package resource
>>>>> is called before any of the openstack resources. I could see this
>>>>> working as an opt-in thing, but not as a default, for the same reason we
>>>>> don't require our users to install pip libraries - there is less
>>>>> security guarantees from pypi and rubygems than from distro packages,
>>>>> plus corporate infrastructure may not allow pulling packages from these
>>>>> types of sources. (I don't see this policy documented anywhere, this was
>>>>> just something that's been instilled in me since I started working on
>>>>> this team. If we want to revisit it, formalize it, or drop it, we can
>>>>> start a separate thread for that.)
>>>>>
>>>>> 2) Vendor the gem as a module. This was how we tried it before, and you
>>>>> can see aimonb/aviator <https://github.com/aimonb/puppet_aviator> for
>>>>> this. The problems I see with this are that we have to keep the module
>>>>> in sync with the gem, and there is no way to keep proper attribution in
>>>>> the module for work that was really done in the gem.
>>>>>
>>>>> I am not convinced that the fork/execs are really causing that much
>>>>> performance issues, though I've heard from some people that
>>>>> python-openstackclient itself is sort of slow.
>>>> That sounds contradictory.
>>>>
>>>> Adding to previous comments on this matter:
>>>>
>>>> Using a native Ruby library (A) to call class any Openstack service's API:
>>>>
>>>> Puppet Agent (Ruby) <=> Net::HTTP (Ruby) <=> Openstack API (HTTP/REST)
>>>>
>>>> Is not comparable with a system call to OSC (B):
>>>>
>>>> Puppet Agent (Ruby) <=> Spawn (System) <=> OpenStackClient (Python) <=>
>>>> Openstack service API (Python)
>>>>
>>>> In the second case, what we're currently using, the spawn is obviously
>>>> bad, although having OSC natively calling Openstack APIs is good.
>>>>
>>>> As said previously, approach A makes effectively more sense.
>>>> Especially from an architecture viewpoint, Puppet providers must be
>>>> dealing with API not CLI.
>>>>
>>>> So do we really need to see metrics comparing A vs B?
>>>> Fuel seems to have a case already anyway.
>>>> But before we jump on a looking glass, structurally, I think, there is
>>>> no doubt.
>>>>
>>>>> The puppet agent itself
>>>>> has a lot of overhead that causes it to be slow, this is why puppetlabs
>>>>> is planning to rewrite it in C++.
>>>> Besides the need to optimize Puppet agent, C++ is not going to change
>>>> anything about the need to call Openstack APIs somehow.
>>>>
>>>> BTW, if it was just me, I would rather use a functional language of the
>>>> past and the future, such as Elixir! ;)
>>>>
>>>>
>>>>> For pro #4, the example you gave isn't really valid because the swift
>>>>> module isn't using openstackclient, and openstackclient has the ability
>>>>> to output data in CSV or JSON. A valid and related complaint would be
>>>>> that openstackclient sometimes spits out ugly irrelevant warnings that
>>>>> we have to parse around, since puppet is terrible at splitting stdout
>>>>> from stderr.
>>>>>
>>>>> I think a major issue we had the first time we tried to do this was that
>>>>> we started doing work to rewrite these things without actually knowing
>>>>> if it would be better. Instead of using that as a precedent for doing
>>>>> the same thing again, I would like to take that experience and do this
>>>>> right by actually gathering data: what is wrong with how we're doing it
>>>>> now, what numbers and metrics show that things are bad? Can we create a
>>>>> proof-of-concept that demonstrates that we can really improve these numbers?
>>>>>
>>>>> If we decide that using a ruby library over the CLI is really a
>>>>> measurably better approach, I would recommend we get in contact with
>>>>> Mark Maglana, the author of Aviator, and get the library moved back over
>>>>> to openstack rather than GitHub, and set up a core reviewer team with
>>>>> multiple members from multiiple companies. We should also work with the
>>>>> OpenStack-SDK team to see if we can get this moved under their project
>>>>> or if they have suggestions for us.
>>>>>
>>>> So if a native Ruby library makes sense, which one? [1]
>>>>
>>>> Before contacting Aviator's author(s) to make it moving or to clone it
>>>> under Openstack umbrella (under th name Astronaut!), we must answer the
>>>> question: Do we really need Aviator?
>>>>
>>>> Aviator is a wrapper of Faraday [2], which itself is wrapping Net::HTTP.
>>>> Faraday which is great for API consumers like us because it allows:
>>>>
>>>> 1. To use different adapters
>>>> ----------------------------
>>>> The default adapter is Net::HTTP.
>>>> Other adapter such as Typhoeus which can perform many requests in
>>>> parallel could be an advantage for example when scaling up.
>>>> A previously mentioned use case scenario of deploying hundreds of users,
>>>> projects and roles.
>>>>
>>>> Although from the puppet agent, to be able to kick such parallel
>>>> creations, we might have to code specifically towards it.
>>>>
>>>> Sure this might looming away with Keystone Federation and 3rd parties
>>>> identity managers, it still remains an issue when Keystone is used as
>>>> the identity service.
>>>>
>>>> 2. The middleware paradigm
>>>> --------------------------
>>>> Openstack Puppet is a *big* API client consumer.
>>>> Look here for more details [3]
>>>>
>>>> This feature would be very beneficial for Openstack Puppet.
>>>>
>>>>
>>>> So, what are the advantage of Aviator?
>>>>
>>>> Besides the Logger, Aviator doesn't seems to be using much of the
>>>> middleware feature.
>>>>
>>>> Sure the Describer makes things look good and I tend to agree with
>>>> Aviator approach from an OO viewpoint.
>>>>
>>>> But that's not enough to hold us with Aviator.
>>>> Please add here if you find anything else.
>>>>
>>>> Don't get me wrong, I think Aviator is a great idea but because of it's
>>>> state and our needs we might just use Faraday and add our own middleware
>>>> wrappers. Those can simply be on a per puppet module basis.
>>>>
>>>> That makes sense from an architecture angle and should give us room for
>>>> scalabitity.
>>>>
>>>> [1] http://www.slideshare.net/HiroshiNakamura/rubyhttp-clients-comparison
>>>> [2] https://github.com/lostisland/faraday
>>>> [3] http://mislav.uniqpath.com/2011/07/faraday-advanced-http/
>>>>
>>> I spent a bit of time on this subject because it always made sense to
>>> me, thanks to Vladimir to bring it back to the table.
>>>
>>> I actually think we don't really need Faraday at all and we could easily
>>> go with Net::HTTP only.
>>>
>>> Because, the 2 main advantages I could see from Faraday, are the HTTP
>>> error propagation and the middleware.
>>> But Net::HTTP provides those HTTP errors and we don't really need
>>> advanced middleware features for HTTP response processing.
>>> Actually the values returned by Openstack APIs are easy to process, from
>>> JSON structures they get transformed directly in Hashes.
>> But you can have middleware for request as well. An interesting
>> middleware I could think of was for authentication. Having all the
>> authentication nicely wrapped in a middleware would be nice.
>>
> +1
Is someone going to port python-keystonemiddleware to ruby, including
all of the supporting code for gssapi/kerberos, federation, etc.?
>
>> You can also have different backend. It could offer a interesting way
>> to offer backward compatibility. Maybe we could define a openstack cli
>> backend that would take exactly the same argument as the current one
>> (daydreaming I think ...)
>>
> Good idea too.
>
>> Anyway, I still think that faraday should be the way to go, but the
>> argument of less dependencies may be compeling, but can be discussed. I
>> think that the client would be anyway distributed as a gem and then
>> packaged. So adding faraday in the mixt wouldn't be that bad.
>>
> Using Faraday would be good.
> Faraday is a quite common gem, it's available on Debian and Fedora but
> it's not packaged yet for Centos/RHEL.
> I believe we don't really need to depend upon any other library, at
> least initially (Read on).
>
>>> Anyway in order to get some of those real data I thought about comparing
>>> efficiency between the OSC and the native approaches.
>>> So I tested the execution time of a manifest creating 1000 tenants using
>>> either approach.
>>>
>>> The result are actually beyond expectation, it's *enormous* (I couldn't
>>> believe when I ran it), it's almost 70 times faster:
>>> - 1000 tenants created using OSC:
>>> real 7m54.582s
>>> user 6m27.764s
>>> sys 0m55.741s
>>>
>>> - 1000 tenants created using native Ruby lib (Net::HTTP):
>>> real 0m40.079s
>>> user 0m8.908s
>>> sys 0m0.799s
>>>
>>> The Ruby native library can scale better!
>> I think that most of the difference here is because the cli is forced to
>> do authentication for each request. I did a simple test on a vm:
>>
>> sysdig 'proc.name=openstack' -c proc_exec_time &
>>
>> ruby -e 'puts "\nservice list"*200' | openstack --os-username beaker-civ3 --os-password secret --os-project-name servicesv3 --os-user-domain-name service_domain --os-project-domain-name service_domain --os-auth-url http://127.0.0.1:5000/v3 --os-identity-api-version 3
>>
>> gives 7.21s,
>>
>> but:
>>
>> eval `ruby -e 'puts "openstack --os-username beaker-civ3 --os-password secret --os-project-name servicesv3 --os-user-domain-name service_domain --os-project-domain-name service_domain --os-auth-url http://127.0.0.1:5000/v3 --os-identity-api-version 3 service list;"*200'`
>>
>> gave ~800ms*200 = 160s
>>
> The authentication absolutely a *killer* but tha's only when not using
> the admin token. The token admin is always used when we deploy openstack
> (bootstrapping).
Only for puppet-keystone. Other modules (e.g. puppet-glance) do not use
token auth.
> Using the admin token means no authentication is done
> since we have a token already.
>
> In my test I was using the token admin too.
>
> That said, the way things were implemented when introducing OSC was to
> offer flexibility and backward compatibility but we have room for
> optimization.
>
>
>> The main difference is due to keystone authentication. One way to solve
>> this is by caching authentication. I don't think you can pass a token
>> to the openstack cli but that would be nice.
> It's the exact same case as when using the admin (or service) token,
> using OS_TOKEN and OS_URL.
>
> This is a good idea and we should definitely do it.
>
>
>> Another way would be to use the first form and instrumentatilize
>> openstackcli using Ruby::PTY and expect. It doesn't look very exiting
>> but that would improve performance by a order of magnitude (22X in my
>> example), without changing the interface at all for the user (without
>> changing much actually)
>>
>> Moving from the cli to dedicated client may add support for richer error
>> code as this is currently tighly bound to what the cli has to offer.
>> But maybe the cli could be improved instead of making a brand new
>> client?
>>
>> I red a mention of fog.io. It's a very nice framework, it supports
>> openstack since 2011 and is very hackable! The problem there, is that
>> it supports a lot of other thing too. So in the spirit of keeping the
>> requirements small, I think this should be excluded from the choice
>> (unfortunatly, as it's the framework I'm most familiar with)
>>
> Agreed.
>
>> So I have mixed feeling on this one. On one side I would be very
>> excited to participate to such a development, on the other side we may
>> reinvent the wheel without having explored simpler solution:
>> - improve openstack cli to support token (or the "server" mode Richard
>> talked about) or whatever authentication caching mecanism;
>> - having an option in the openstack cli to return the exact server
>> error (code and message);
>> - maybe instrumentatlize using PTY/expect to gain speed without
>> changing anything to the interface.
>
> The current approach using OSC must be fixed by fetching and storing the
> token in @credentials object.
>
> Again, using Openstack Puppet modules outside of deployments represents
> only only a portion of the use cases.
>
> Therefore the native client approach question is still relevant.
> Efficiency using API will be
> And structurally it's not going away anytime soon.
>
> Avoiding any dependency is doable and probably a better path, for now,
> if we decide to take the API road, because we can have a minimal impact too.
>
> Here is a Net::HTTP implementation base along the keystone_tenant
> modified to use it:
> https://review.openstack.org/238750
> https://review.openstack.org/238752
>
>
>> Cheers,
>>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list