[openstack-dev] [keystone] Pagination

Jay Pipes jaypipes at gmail.com
Tue Aug 13 17:57:38 UTC 2013


On 08/13/2013 01:43 PM, Henry Nash wrote:
> Jay,
>
> Thanks for all the various links - most useful.

No problem. It's an old problem in the OpenStack world that's been 
discussed many times before :) Might as well take advantage of prior 
art/discussions... I personally had a change of heart on the issue a 
couple years ago after the initial discussions.

> To map this into keystone context, if we were to follow this logic we would:
>
> 1) Support 'limit' and 'marker' (as opposed to 'page', 'page_szie', or anything else).  These would be standard, independent of what backing store keystone was using.  If neither are included in the url, then we return the first N entires, where N is defined by the cloud provider.  This ensures that for at least smaller deployments, non-pagination aware clients still work.  If either 'limit' or 'marker' are specified, then we paginate, passing them down into the driver layer wherever possible to ensure efficiency (some drivers may not be able to support pagination, hence we will do this, inefficiently, at a higher layer)
> 2) If we are paginating at the driver level, we must, by definition, be doing all the filtering down there as well (otherwise it all gets mucked)
> 3) We should look at supporting the other standard options (sort order etc.), but irrespective of that, by definition, we must ensure that we any driver that is paginating must be getting is entries back in a consistent order (otherwise, again, pagination doesn't work reliably)

Yup, all of the above meets matches my understanding. Filter support 
should come first, followed by paging-parameter pushdown to the engines.

Best,
-jay

> On 13 Aug 2013, at 18:10, Jay Pipes wrote:
>
>> On 08/13/2013 12:55 PM, Lyle, David (Cloud Services) wrote:
>>> The marker/limit pagination scheme is inferior.
>>
>> A bold statement that flies in the face of experience and the work already done in all the other projects.
>>
>>> The use of page/page_size allows access to arbitrary pages, whereas limit/marker only allows forward progress.
>>
>> I don't see this as a particularly compelling use case considering the performance manifestations of using LIMIT OFFSET pagination.
>>
>>> In Horizon's use case, with page/page_size we can provide the user access to any page they have already visited, rather than just the previous page (using prev/next links returned in the response).
>>
>> I don't see this as a particularly useful thing, but in any case, you could still do this by keeping the markers for previous pages on the client (Horizon) side.
>>
>> The point of marker/limit is to eliminate poor performance of LIMIT OFFSET queries and to force proper index usage in the listing queries.
>>
>> You can see the original discussion about this from more than two years and even see where I was originally arguing for a LIMIT OFFSET strategy and was brought around to the current limit/marker strategy by the responses of Justin Santa Barbara and Greg Holt:
>>
>> https://lists.launchpad.net/openstack/msg02548.html
>>
>> Best,
>> -jay
>>
>>> -David
>>>
>>> On 08/13/2013 10:29 AM, Pipes, Jay wrote:
>>>
>>>> On 08/13/2013 03:05 AM, Yee, Guang wrote:
>>>>> Passing the query parameters, whatever they are, into the driver if
>>>>> the given driver supports pagination and allowing the driver to
>>>>> override the manager default pagination functionality seem reasonable to me.
>>>
>>>> Please do use the standards that are supported in other OpenStack services already: limit, marker, sort_key and sort_dir.
>>>
>>>> Pagination is meaningless without a sort key and direction, so picking a sensible default for user/project records is good. I'd go with either created_at (what Glance/Nova/Cinder use..) or with the user/project >UUID.
>>>
>>>> The Glance DB API pagination is well-documented and clean [1]. I highly recommend it as a starting point.
>>>
>>>> Nova uses the same marker/limit/sort_key/sort_dir options for queries that it allows pagination on. An example is the
>>>> instance_get_all_by_filters() call [2].
>>>
>>>> Cinder uses the same marker/limit/sort_key/sort_dir options for query pagination as well. [3]
>>>
>>>> Finally, I'd consider supporting the standard change-since parameter for listing operations. Both Nova [4] and Glance [5] support the parameter, which is useful for tools that poll the APIs for "new" >events/records.
>>>
>>>> In short, go with what is already a standard in the other projects...
>>>
>>>> Best,
>>>> -jay
>>>
>>>> [1]
>>>> https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L429
>>>> [2]
>>>> https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1709
>>>> [3]
>>>> https://github.com/openstack/cinder/blob/master/cinder/common/sqlalchemyutils.py#L33
>>>> [4]
>>>> https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1766
>>>> [5]
>>>> https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L618
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>




More information about the OpenStack-dev mailing list