Open Stack

On 17/11/16 19:01, Monty Taylor wrote:
> On 11/17/2016 05:24 PM, Zane Bitter wrote:
>> On 15/11/16 09:56, Monty Taylor wrote:
>>> Hey everybody!
>>>
>>> At this past OpenStack Summit the results of the Interop Challenge were
>>> shown on stage. It was pretty awesome - 17 different people from 17
>>> different clouds ran the same workload. And it worked!
>>>
>>> However, one of the reasons it worked is because they all used the
>>> Ansible modules we wrote that are based on the shade library that
>>> contains the business logic needed to hide vendor differences in clouds.
>>> That means that there IS a fantastic OpenStack interoperability story -
>>> but only if you program in Python. That's less awesome.
>>
>> So I don't want to criticise this effort, because I'm sure that it's
>> very valuable and worthy &c.
>>
>> But it does make me sad that we've so thoroughly given up on the concept
>> of making the OpenStack APIs themselves interoperable that we're
>> building an API for our APIs (Yo dawg!) to work around it.
>
> Tell me about it. I share your sadness. In fact, that sadness has been
> my main state for several years now.
>
>> The problem is that to take advantage of the interoperability benefits
>> you'll be locked in to a single orchestration tool (Ansible/shade). If
>> you have a particular reason to use another tool (possibly, ahem, the
>> one that is an official part of OpenStack and already available in 2/3
>> of OpenStack clouds... but also Puppet, JuJu, &c.) then you'll have to
>> choose between whatever feature you wanted there and interoperability.
>> That's taking "there IS a fantastic OpenStack interoperability story -
>> but only if you program in Python" and kicking the can one step down the
>> road (s/program in Python/orchestrate in Ansible). Whereas if we fix the
>> underlying APIs then *everyone* benefits.
>
> I seem to have communicated something very poorly if that's the take
> away you've gotten. Today, if you want cross-cloud interop, you pretty
> much need to use shade/ansible. That's not cool - because it's limiting
> to people wanting to work in python and people wanting to orchestrate in
> ansible. It should absolutely not be necessary to want to use those
> tools to get the interop story.
>
> So one of the main reasons to do this is precisely to provide that story
> to everyone - whether they're doing Juju and go, or puppet and ruby, or
> just programming in Fog or whatnot.

So, say that I want to create my servers in Heat so that I can use Heat 
software deployments for orchestration. How would I go about e.g. making 
sure that the servers are always connected to the networks I expect on a 
variety of different clouds? Would Oaktree figure out the networks and 
pass them in to the stack when creating it? (I don't believe shade has 
Heat support yet, but we should definitely add it & I don't foresee any 
great obstacle.) Or would Heat have to add Oaktree resource types?

It sounds to me like the former approach would require all of the same 
work in the template that you'd need to handle it now (using 
conditionals), and the only real difference is that instead of providing 
your own environment file for each cloud you do a bit of oaktree 
integration and that figures it out for you instead.

Adding Oaktree resource types to Heat to paper over isn't really a 
solution from my point of view.

>> I feel like the entire OpenStack project has, out of a desire not to be
>> opinionated, consistently failed both our users and operators by
>> encouraging all sorts of unnecessarily incompatible configurations. Not
>> to pick on any particular project but e.g. can anyone tell me why
>> Neutron doesn't automatically come, out of the box, with external
>> networks called "internet" and "openstack" so that users can create
>> floating IPs that talk to either the internet or just the control plane,
>> respectively, on any OpenStack cloud with a single Heat template (or
>> whatever) without having to paste UUIDs anywhere? What sane reason could
>> there be to even allow, let alone force, all operators to solve these
>> problems independently?
>>
>> I'm sure the infra team can think of 100 pet annoyances like that. So
>> carry on, but maybe y'all could make a list somewhere of all the
>> interoperability problems that shade has had to work around and we could
>> try to make it a priority as a community to address them?
>
> Absolutely happy to. Some are in these slides:
>
> http://inaugust.com/talks/real-slim-shade.html
>
> (http://inaugust.com/talks/real-slim-shade.html#/32 is a good slide to
> start with)
>
> But I'd be happy to list out a set of issues in a form that doesn't
> involve watching me give a talk.
>
>>
>>> With that in mind - I'm pleased to announce a new project that aims to
>>> address that - oaktree.
>>>
>>> oaktree is a gRPC-based API porcelain service for OpenStack that is
>>> based on the shade library and I'd love some help in writing it.
>>>
>>> Basing oaktree on shade gets not only the business logic. Shade already
>>> understands a multi-cloud world. And because we use shade in Infra for
>>> nodepool, it already has caching, batching and thundering herd
>>> protection sorted to be able to hand very high loads efficiently. So
>>> while oaktree is new, the primary logic and fundamentals are all shade
>>> and are battle-tested.
>>>
>>> The barrier to deployers adding it to their clouds needs to be as low as
>>> humanly possible. So as we work on it, ensuring that we keep it
>>> dead-simple to install, update and operate must be a primary concern.
>>>
>>> Where are we and what's next?
>>>
>>> oaktree doesn't do a whole lot that's terribly interesting at the
>>> moment. We have all of the development scaffolding and gate jobs set up
>>> and a few functions implemented.
>>>
>>> oaktree exists currently as two repos - oaktree and oaktreemodel:
>>>
>>>   http://git.openstack.org/cgit/openstack/oaktree
>>>   http://git.openstack.org/cgit/openstack/oaktreemodel
>>>
>>> oaktreemodel contains the Protobuf definitions and the build scripts to
>>> produce Python, C++ and Go code from them. The python code is published
>>> to PyPI as a normal pure-python library. The C++ code is published as a
>>> source tarball and the Go code is checked back in to the same repo so
>>> that go works properly.
>>>
>>> oaktree depends on the python oaktreemodel library, and also on shade.
>>> It implements the server portion of the gRPC service definition.
>>>
>>> Currently, oaktree can list and search for flavors, images and floating
>>> ips. Exciting right? Most of the work to expose the rest of the API that
>>> shade can provide at the moment is going to be fairly straightforward -
>>> although in each case figuring out the best mapping will take some care.
>>>
>>> We have a few major things that need some good community design. These
>>> are also listed in a todo.rst file in the oaktree repo which is part of
>>> the docs:
>>>
>>>   http://oaktree.readthedocs.io/en/latest/
>>>
>>> The auth story. The native/default auth for gRPC is oauth. It has the
>>> ability for pluggable auth, but that would raise the barrier for new
>>> languages. I'd love it if we can come up with a story that involves
>>> making API users in keystone and authorizing them to use oaktree via an
>>> oauth transaction. The keystone auth backends currently are all about
>>> integrating with other auth management systems, which is great for
>>> environments where you have a web browser, but not so much for ones
>>> where you need to put your auth credentials into a file so that your
>>> scripts can work. I'm waving my hands wildly here - because all I really
>>> have are problems to solve and none of the solutions I have are great.
>>>
>>> Glance Image Uploads and Swift Object Uploads (and downloads). Having
>>> those two data operations go through an API proxy seems inefficient.
>>> However, having them not in the API seems like a bad user experience.
>>> Perhaps if we take advantage of the gRPC streaming protocol support
>>> doing a direct streaming passthrough actually wouldn't be awful. Or
>>> maybe the better approach would be for the gRPC call to return a URL and
>>> token for a user to POST/PUT to directly. Literally no clue.
>>>
>>> In any case - I'd love help from anyone who thinks this sounds like a
>>> good idea. In a perfect world we'll have something ready for 1.0 by
>>> Atlanta.
>>>
>>> Join us in #openstack-shade if you want to hack.
>>>
>>> Thanks!
>>> Monty
>>>
>>>
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>