Open Stack

Tihomir,

Today I added one glance call based on Richard’s decorator pattern[1] and started to play with incorporating some of your ideas. Please note, I only had limited time today.  That is passing the kwargs through to the glance client. This was an interesting first choice, because it immediately highlighted a concrete example of the horizon glance wrapper post-processing still being useful (rather than be a direct pass-through with no wrapper). See below. If you have some some concrete code examples of your ideas it would be helpful.

[1] https://review.openstack.org/#/c/141273/2/openstack_dashboard/api/rest/glance.py

With the patch, basically, you can call the following and all of the GET parameters get passed directly through to the horizon glance client and you get results back as expected.

http://localhost:8002/api/glance/images/?sort_dir=desc&sort_key=created_at&paginate=True&marker=bb2cfb1c-2234-4f54-aec5-b4916fe2d747

If you pass in an incorrect sort_key, the glance client returns the following error message which propagates back to the REST caller as an error with the message:

"sort_key must be one of the following: name, status, container_format, disk_format, size, id, created_at, updated_at."

This is done by passing **request.GET.dict() through.

Please note, that if you try this (with POSTMAN, for example), you need to set the header of X-Requested-With = XMLHttpRequest

So, what issues did it immediately call out with directly invoking the client?

The python-glanceclient internally handles pagination by returning a generator.  Each iteration on the generator will handle making a request for the next page of data. If you were to just do something like return list(image_generator) to serialize it back out to the caller, it would actually end up making a call back to the server X times to fetch all pages before serializing back (thereby not really paginating). The horizon glance client wrapper today handles this by using islice intelligently along with honoring the API_RESULT_LIMIT setting in Horizon. So, this gives a direct example of where the wrapper does something that a direct passthrough to the client would not allow.

That said, I can see a few ways that we could use the same REST decorator code and provide direct access to the API.  We’d simply provide a class where the url_regex maps to the desired path and gives direct passthrough. Maybe that kind of passthrough could always be provided for ease of customization / extensibility and additional methods with wrappers provided when necessary.  I need to leave for today, so can’t actually try that out at the moment.

Thanks,
Travis

From: Thai Q Tran <tqtran at us.ibm.com<mailto:tqtran at us.ibm.com>>
Reply-To: OpenStack List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Friday, December 12, 2014 at 11:05 AM
To: OpenStack List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] [horizon] REST and Django


In your previous example, you are posting to a certain URL (i.e. /keystone/{ver:=x.0}/{method:=update}).
<client: POST /keystone/{ver:=x.0}/{method:=update}> => <middleware: just forward to clients[ver].getattr("method")(**kwargs)> => <keystone: update>

Correct me if I'm wrong, but it looks like you have a unique URL for each /service/version/method.
I fail to see how that is different than what we have today? Is there a view for each service? each version?

Let's say for argument sake that you have a single view that takes care of all URL routing. All requests pass through this view and contain a JSON that contains instruction on which API to invoke and what parameters to pass.
And lets also say that you wrote some code that uses reflection to map the JSON to an action. What you end up with is a client-centric application, where all of the logic resides client-side. If there are things we want to accomplish server-side, it will be extremely hard to pull off. Things like caching, websocket, aggregation, batch actions, translation, etc.... What you end up with is a client with no help from the server.

Obviously the other extreme is what we have today, where we do everything server-side and only using client-side for binding events. I personally prefer a more balance approach where we can leverage both the server and client. There are things that client can do well, and there are things that server can do well. Going the RPC way restrict us to just client technologies and may hamper any additional future functionalities we want to bring server-side. In other words, using REST over RPC gives us the opportunity to use server-side technologies to help solve problems should the need for it arises.

I would also argue that the REST approach is NOT what we have today. What we have today is a static webpage that is generated server-side, where API is hidden from the client. What we end up with using the REST approach is a dynamic webpage generated client-side, two very different things. We have essentially striped out the rendering logic from Django templating and replaced it with Angular.


-----Tihomir Trifonov <t.trifonov at gmail.com<mailto:t.trifonov at gmail.com>> wrote: -----
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
From: Tihomir Trifonov <t.trifonov at gmail.com<mailto:t.trifonov at gmail.com>>
Date: 12/12/2014 04:53AM
Subject: Re: [openstack-dev] [horizon] REST and Django

Here's an example: Admin user Joe has an Domain open and stares at it for 15 minutes while he updates the description. Admin user Bob is asked to go ahead and enable it. He opens the record, edits it, and then saves it. Joe finished perfecting the description and saves it. Doing this action would mean that the Domain is enabled and the description gets updated. Last man in still wins if he updates the same fields, but if they update different fields then both of their changes will take affect without them stomping on each other. Whether that is good or bad may depend on the situation…


That's a great example. I believe that all of the Openstack APIs support PATCH updates of arbitrary fields. This way - the frontend(AngularJS) can detect which fields are being modified, and to submit only these fields for update. If we however use a form with POST, although we will load the object before updating it, the middleware cannot find which fields are actually modified, and will update them all, which is more likely what PUT should do. Thus having full control in the frontend part, we can submit only changed fields. If however a service API doesn't support PATCH, it is actually a problem in the API and not in the client...



The service API documentation almost always lags (although, helped by specs now) and the service team takes on the burden of exposing a programmatic way to access the API.  This is tested and easily consumable via the python clients, which removes some guesswork from using the service.

True. But what if the service team modifies a method signature from let's say:

def add_something(self, request,
​ field1, field2):


to

def add_something(self, request,
​ field1, field2, field3):


and in the middleware we have the old signature:

​def add_something(self, request,
​ field1, field2):

we still need to modify the middleware to add the new field. If however the middleware is transparent and just passes **kwargs, it will pass through whatever the frontend sends. So we just need to update the frontend, which can be done using custom views, and not necessary going through an upstream change. My point is why do we need to hide some features of the backend service API behind a "firewall" what the middleware in fact is?







On Fri, Dec 12, 2014 at 8:08 AM, Tripp, Travis S <travis.tripp at hp.com<mailto:travis.tripp at hp.com>> wrote:
I just re-read and I apologize for the hastily written email I previously
sent. I’ll try to salvage it with a bit of a revision below (please ignore
the previous email).

On 12/11/14, 7:02 PM, "Tripp, Travis S" <travis.tripp at hp.com<mailto:travis.tripp at hp.com>> wrote
(REVISED):

>Tihomir,
>
>Your comments in the patch were very helpful for me to understand your
>concerns about the ease of customizing without requiring upstream
>changes. It also reminded me that I’ve also previously questioned the
>python middleman.
>
>However, here are a couple of bullet points for Devil’s Advocate
>consideration.
>
>
>  *   Will we take on auto-discovery of API extensions in two spots
>(python for legacy and JS for new)?
>  *   The Horizon team will have to keep an even closer eye on every
>single project and be ready to react if there are changes to the API that
>break things. Right now in Glance, for example, they are working on some
>fixes to the v2 API (soon to become v2.3) that will allow them to
>deprecate v1 somewhat transparently to users of the client library.
>  *   The service API documentation almost always lags (although, helped
>by specs now) and the service team takes on the burden of exposing a
>programmatic way to access the API.  This is tested and easily consumable
>via the python clients, which removes some guesswork from using the
>service.
>  *   This is going to be an incremental approach with legacy support
>requirements anyway.  So, incorporating python side changes won’t just go
>away.
>  *   Which approach would be better if we introduce a server side
>caching mechanism or a new source of data such as elastic search to
>improve performance? Would the client side code have to be changed
>dramatically to take advantage of those improvements or could it be done
>transparently on the server side if we own the exposed API?
>
>I’m not sure I fully understood your example about Cinder.  Was it the
>cinder client that held up delivery of horizon support, the cinder API or
>both?  If the API isn’t in, then it would hold up delivery of the feature
>in any case. There still would be timing pressures to react and build a
>new view that supports it. For customization, with Richard’s approach new
>views could be supported by just dropping in a new REST API decorated
>module with the APIs you want, including direct pass through support if
>desired to new APIs. Downstream customizations / Upstream changes to
>views seem a bit like a bit of a related, but different issue to me as
>long as their is an easy way to drop in new API support.
>
>Finally, regarding the client making two calls to do an update:
>
>​>>Do we really need the lines:​
>
>>> project = api.keystone.tenant_get(request, id)
>>> kwargs = _tenant_kwargs_from_DATA(request.DATA, enabled=None)
>​
>I agree that if you already have all the data it may be bad to have to do
>another call. I do think there is room for discussing the reasoning,
>though.
>As far as I can tell, they do this so that if you are updating an entity,
>you have to be very specific about the fields you are changing. I
>actually see this as potentially a protectionary measure against data
>loss and sometimes a very nice to have feature. It perhaps was intended
>to *help* guard against race conditions (no locking and no transactions
>with many users simultaneously accessing the data).
>
>Here's an example: Admin user Joe has a Domain open and stares at it for
>15 minutes while he updates just the description. Admin user Bob is asked
>to go ahead and enable it. He opens the record, edits it, and then saves
>it. Joe finished perfecting the description and saves it. They could in
>effect both edit the same domain independently. Last man in still wins if
>he updates the same fields, but if they update different fields then both
>of their changes will take affect without them stomping on each other. Or
>maybe it is intended to encourage client users to compare their current
>and previous to see if they should issue a warning if the data changed
>between getting and updating the data. Or maybe like you said, it is just
>overhead API calls.