Open Stack

On 05/19/2017 01:04 PM, Sean Dague wrote:
> On 05/19/2017 01:38 PM, Dean Troyer wrote:
>> On Fri, May 19, 2017 at 11:53 AM, Chris Friesen
>> <chris.friesen at windriver.com> wrote:
>>> ..., but it seems to me that the logical
>>> extension of that is to expose simple orthogonal APIs where the nova boot
>>> request should only take neutron port ids and cinder volume ids.  The actual
>>> setup of those ports/volumes would be done by neutron and cinder.
>>>
>>> It seems somewhat arbitrary to say "for historical reasons this subset of
>>> simple things can be done directly in a nova boot command, but for more
>>> complicated stuff you have to go use these other commands".  I think there's
>>> an argument to be made that it would be better to be consistent even for the
>>> simple things.
>>
>> cdent mentioned enamel[0] above, and there is also oaktree[1], both of
>> which are wrapper/proxy services in front of existing OpenStack APIs.
>> I don't know enough about enamel yet, but one of the things I like
>> about oaktree is that it is not required to be deployed by the cloud
>> operator to be useful, I could set it up and proxy Rax and/or
>> CityCloud and/or mtreinish's closet cloud equally well.
>>
>> The fact that these exist, and things like shade itself, are clues
>> that we are not meeting the needs of API consumers.  I don't think
>> anyone disagrees with that; let me know if you do and I'll update my
>> thoughts.
>
> It's fine to have other ways to consume things. I feel like "make
> OpenStack easier to use by requiring you install a client side API
> server for your own requests" misses the point of the easier part. It's
> cool you can do it as a power user. It's cool things like Heat exist for
> people that don't want to write API calls (and just do templates). But
> it's also not helping on the number of pieces of complexity to manage in
> OpenStack to have a workable cloud.

Yup. Agree. Making forward progress on that is paramount.

> I consider those things duct tape, leading use to the eventually
> consistent place where we actually do that work internally. Because,
> having seen with the ec2-api proxy, the moment you get beyond trivial
> mapping, you now end up with a complex state tracking system, that's
> going to need to be highly available, and replicate a bunch of your data
> to be performent, and then have inconsistency issues, because a user
> deployed API proxy can't have access to the notification bus, and... boom.

You can actually get fairly far (with a few notable exceptions - I'm 
looking at you unattached floating ips) without state tracking. It comes 
at the cost of more API spidering after a failure/restart. Being able to 
cache stuff aggressively combined with batching/rate-limiting of 
requests to the cloud API allows one to do most of this to a fairly 
massive scale statelessly. However, caching, batching and rate-limiting 
are all pretty much required else you wind up crashing public clouds. :)

I agree that the things are currently duct tape, but I don't think that 
has to be a bad thing. The duct tape is currently needed client-side no 
matter what we do, and will be for some time no matter what we do 
because of older clouds. What's often missing is closing the loop so 
that we can, as OpenStack, eventually provide out of the box the consume 
experience that people currently get from using one of the client-side 
duct tapes. That part is harder, but it's definitely important.

> You end up replicating the Ceilometer issue where there was a break down
> in getting needs expressed / implemented, and the result was a service
> doing heavy polling of other APIs (because that's the only way it could
> get the data it needed). Literally increasing the load on the API
> surfaces by a factor of 10

Right. This is why communication is essential. I'm optimistic we can do 
well on this topic, because we are MUCH better are talking to each other 
now than we were back when ceilometer was started.

Also, a REST-consuming porcelain like oaktree gets to draw on real-world 
experience consuming OpenStack's REST APIs at scale. So it's also not 
the same problem setup, since it's not a from-scratch new thing.

This is, incidentally, why experience with caching and batching is 
important. There is a reason why we do GET /servers/detail once every 5 
seconds rather than doing a specific GET /server/{id}/detail calls for 
each booting VM.

Look at what we could learn just from that... Users using shade are 
doing a full detailed server list because it scales better for 
concurrency. It's obviously more expensive on a single-call basis. BUT - 
maybe it's useful information that doing optimization work on GET 
/servers/detail could be beneficial.

> (http://superuser.openstack.org/articles/cern-cloud-architecture-update/
> last graph). That was an anti pattern. We should have gotten to the
> bottom of the mismatches and communication issues early on, because the
> end state we all inflicted on users to get a totally reasonable set of
> features, was not good. Please lets not do this again.

++

> These should be used as ways to experiment with the kinds of interfaces
> we want cheaply, then take them back into services (which is a more
> expensive process involving compatibility stories, deeper documentation,
> performance implications, and the like), not an end game on their own.

Yup. Amongst other things this is why I'm pushing so hard on service and 
version discovery right now. I can do it fully and correctly in shade- 
but then what about non-shade users? We've still got probably a 2 year 
path to the point where the client logic doesn't need to be duplicated 
everywhere ... but working on that path is super important.

>> First and foremost, we need to have the primitive operations that get
>> composed into the higher-level ones available.  Just picking "POST
>> /server" as an example, we do not have that today.  Chris mentions
>> above the low-level version should take IDs for all of the associated
>> resources and no magic happening behind the scenes.  I think this
>> should be our top priority, everything else builds on top of that, via
>> either in-service APIs or proxies or library wrappers, whatever a) can
>> get implemented and b) makes sense for the use case.
>
> You can get the behavior. It also has other behaviors. I'm not sure any
> user has actually argued for "please make me do more rest calls to
> create a server".

No. Please no. I'd like to make less calls.

> Anyway, this gets pretty meta pretty fast. I agree with Zane saying "I
> want my server to build", or "I'd like Nova to build a volume for me"
> are very odd things to call PaaS. I think of PaaS as "here is a ruby on
> rails app, provision me a db for it, and make it go". Heroku style.

I'd like for POST /servers to support an auto_ip flag like shade gives 
users now. Then, when it does, I'd love to have shade use it when it's 
there, and fall back to making the 20 billion API calls it takes when 
it's not. get-me-a-network is similar, although in this case nova and 
neutron got to it before shade did. So the shade get-me-a-network plan 
is going to be to add the flag to create_server, but have a fallback 
impl that will make all of the neutron calls itself if the nova in 
question doesn't support get-me-a-network itself.

I mention those for a couple of reasons:

I agree that I think there are some additional 'orchestration' seeming 
things that putting in nova/neutron/cinder, etc would cause the world to 
be a better place

I think it shows a potentially nice pattern - which is a "time machine" 
like function. When there is a thing that can be used by an API consumer 
to get up to date semantics on _their_ timeframe, it gives us a vehicle 
to deliver improvements to OpenStack, but to also empower consumers of 
older clouds.

The spec on service-types-authority and aliases is an example of this 
working as I think should work moving forward. Rather than just 
implementing that in shade and being done with it - we're working to 
define the behavior we need in the future. Then we can implement it in 
shade, and because it's well defined so can other client libs. At a 
point in the future once all of the OpenStack clouds out there have 
adopted the recommendations, all of the clients will be doing less work, 
but they also get their users to the future today.

A porcelain that is only an API consumer (and doesn't touch the message 
bus) like oaktree just extends that idea to people who are not python 
programmers. It also explicitly embodies the time-machine idea.

As developers working on OpenStack, there is a timeline. It includes 
things like deprecation notices and removals. From an OpenStack service 
developer POV the removal of nova-network API support from 
python-novaclient makes perfect sense, because nova knows that the Pike 
version of Nova doesn't contain nova-network. Nova can also know that 
the Pike version of Cinder _definitely_ contains cinder v3. Pike Nova 
needs to worry about Ocata Cinder for upgrades, but Pike Nova is 
definitely not worried about Kilo Cinder. Why would it?

But as developers writing libraries and tools to _consume_ OpenStack, 
there is no such timeline. Diablo is as alive and well as Pike and must 
be taken in to account. Our friends at Oracle are running Kilo - we 
don't want the OpenStack Ansible modules to just stop working for them, 
do we? Of course not.

This is why I think an API-consumer porcelain can benefit our users AND 
can benefit us.

A deployer can deploy one directly and expose it to their users. Since 
it's explicitly written to work with old OpenStacks, they can even 
update it frequently without updating the rest of their cloud. There is 
a benefit to the deployer of running one because they can ensure the 
caching and rate-limiting settings are such that they don't get pummeled 
as they might otherwise get his if they had 1000 users each running 
their own.

If a deployer doesn't, the user is empowered to time-machine themselves 
if they want to. It might be slightly less efficient - and a bit 
annoying since they have to run a service, but it'll work if they happen 
to be in a scenario where they need to use an older cloud and they don't 
happen to be a python developer.

It benefits us because we can continue to work on business-logic 
patterns in the time-machine, then continue to learn which of them are 
things we should push down into the service.

Like maybe we wind up driving more Zaqar adoption so that we've got an 
answer for waiting on a server to be done without polling. Or maybe we 
realize there is a new abstraction that neutron needs called "An IP 
Address" - that a user can request and they'll get a floating IP on FIP 
clouds and a disconnected port on clouds where that's a possiblity. And 
then a user can have the same workflow for creating a long-lived IP, 
booting a server and attaching the IP regardless of whether the 
implementation of that IP is NAT or not.

We also have a vehicle to explore entirely new concepts by using gRPC 
and not REST. Active return channels in gRPC that come from http/2 open 
up some _great_ possibilities. Registering callbacks or waiting for 
notifications is just built-in to gRPC in the first place. A person 
writing an application talking to oaktree gets to just say "boot me a 
server and I'm going to wait until it's done" - and there is no 
client-side polling. Of course, oaktree will be polling OpenStack - but 
it'll be doing it the efficient way that almost nobody writing their own 
polling code is going to bother doing but that we did once and now it 
can be used. But maybe we learn that service-to-service communication 
would be improved by a gRPC layer. Or maybe we don't.

Adding a new API feature at the shade/oaktree layer is trivial - at 
least compared to the cost of adding an API feature like 
get-me-a-network to nova and neutron. If it turns out to be a dumb idea, 
keeping it until the end of time is low cost. But if turns out to be 
awesome, then we know that figuring out how to push it down into the 
REST layer is worthwhile.

Anyway - those are reasons I think a porcelain layer isn't necessarily a 
bad idea. I think there's real potential end-user value. I also don't 
think it's the solution to all of life's problems. If the underlying 
REST APIs don't evolve over time to reduce the amount of client logic 
employed on a fresh new and modern cloud then we're ultimately not 
progressing, existence of a porcelain layer or not.

Monty

PS. I spoke of oaktree in the present tense, as it is a thing. But in 
case you haven't been following that and now you're like "oh boy, 
another thing I need to pay attention to!"

We have made a lot of progress in the last six months, but you will not 
notice any of it in the oaktree repos. That is because as we started 
cranking on oaktree in earnest we realized there were two issues in 
shade that needed to be fixed before further work in the oaktree repos 
would be efficient. The first was that, although shade has an API in 
terms of the methods it exposes - the data returned from the API was 
mushy (protobuf likes mushy data much less than python). So we 
implemented data model contracts and docs for them. The other was that 
the python-*client depends in shade hit the tipping point where we 
needed to get rid of them more urgently. It's not an "easy for a user to 
install locally" if it depends on the entire phone book. We're very 
close to being done with replacing shade's use of python-*client with 
direct REST calls via keystoneauth. Once we're done with that, we can 
once again turn attention to making evident forward progress on oaktree 
itself. (I've been pointing people who want to help with oaktree to 
finishing the RESTification)

Which is all to say, the project is still very much a thing. Once 
RESTification is done, I'll follow up with folks on where we are and 
what's next for it.