Open Stack

Hi

So few comebacks to the various comments:

1) While I understand the idea that a client would follow the next/prev links returned in collections, I wasn't aware that we considered 'page'/'per-page' as not standardized.   We list these explicitly throughout the identity API spec (look in each List 'entity' example).  How I imagined it would work would be:

a) If a client did not include 'page' in the url we would not paginate
b) Once we are paginating, a client can either build the next/prevs urls themselves if they want (by incrementing/decrementing the page number), or just follow the next/prev links (which come with the appropriate 'page=x' in them) returned in the collection which saves them having to do this.
c) Regarding implementation, the controller would continue to be able to paginate on behalf of drivers that couldn't, but those paginate-aware drivers would take over that capability (and indicate this to the controller the state of the pagination so that it can build the correct next/prev links)

2) On the subject of huge enumerates, options are:
a) Support a backend manager scoped (i.e. identity/assignent/token) limit in the conf file which would be honored by drivers.  Assuming that you set this larger than your pagination limit, this would make sense whether your driver is paginating or not in terms of minimizing the delay in responding data as well as not messing up pagination.  In the non-paginated case when we hit the limit, should we indicate this to the client?  Maybe a 206 return code?  Although i) not quite sure that meets http standards, and ii) would we break a bunch of clients by doing this?
b) We scrap the whole idea of pagination, and just set a conf limit as in 2a).  To make this work of course, we must implement any defined filters in the backend (otherwise we still end up with today's performance problems - remember that today, in general,  filtering is done in the controller on a full enumeration of the entities in question).  I was planning to implement this backend filtering anyway as part of (or on top of) my change, since we are holding (at least one of) our hands behind our backs right now by not doing so.  And our filters need to be powerful, do we support wildcards for example, e.g. GET /users?name = fred*  ?
 
Henry

On 13 Aug 2013, at 04:40, Adam Young wrote:

> On 08/12/2013 09:22 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis) wrote:
>> The main reason I use user lists (i.e. keystone user-list) is to get the list of usernames/IDs for other keystone commands. I do not see the value of showing all of the users in an LDAP server when they are not part of the keystone database (i.e. do not have roles assigned to them). Performing a “keystone user-list” command against the HP Enterprise Directory locks up keystone for about 1 ½ hours in that it will not perform any other commands until it is done.  If it is decided that user lists are necessary, then at a minimum they need to be paged to return control back to keystone for another command.
> 
> We need a way to tell HP ED to limit the number of rows, and to do filtering.
> 
> We have a bug for the second part.  I'll open one for the limit.
> 
>>  
>> Mark
>>  
>> From: Adam Young [mailto:ayoung at redhat.com] 
>> Sent: Monday, August 12, 2013 5:27 PM
>> To: openstack-dev at lists.openstack.org
>> Subject: Re: [openstack-dev] [keystone] Pagination
>>  
>> On 08/12/2013 05:34 PM, Henry Nash wrote:
>> Hi
>>  
>> I'm working on extending the pagination into the backends.  Right now, we handle the pagination in the v3 controller class....and in fact it is disabled right now and we return the whole list irrespective of whether page/per-page is set in the query string, e.g.:
>> Pagination is a broken concept. We should not be returning lists so long that we need to paginate.  Instead, we should have query limits, and filters to refine the queries.
>> 
>> Some people are doing full user lists against LDAP.  I don't need to tell you how broken that is.  Why do we allow user-list at the Domain (or unscoped level)?  
>> 
>> I'd argue that we should drop enumeration of objects in general, and certainly limit the number of results that come back.  Pagination in LDAP requires cursors, and thus continuos connections from Keystone to LDAP...this is not a scalable solution.
>> 
>> Do we really need this?
>> 
>> 
>> 
>>  
>>     def paginate(cls, context, refs):
>>         """Paginates a list of references by page & per_page query strings."""
>>         # FIXME(dolph): client needs to support pagination first
>>         return refs
>>  
>>         page = context['query_string'].get('page', 1)
>>         per_page = context['query_string'].get('per_page', 30)
>>         return refs[per_page * (page - 1):per_page * page]
>>  
>> I wonder both for the V3 controller (which still needs to handle pagination for backends that do not support it) and the backends that do....whether we could use wether 'page' is defined in the query-string as an indicator as to whether we should paginate or not?  That way clients who can handle it can ask for it, those that don'twill just get everything.  
>>  
>> Henry
>>  
>> 
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>  
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130813/2f135675/attachment.html>