Open Stack

On 08/13/2013 01:51 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis) 
wrote:
> I have been following this exchange of ideas on how to solve/implement pagination. I would ask you to keep in mind that a solution needs to take into account a split LDAP/SQL backend (you are not always dealing with a single Keystone SQL database). Having a split backend means that the query information is divided between both backends and that you may not have as much flexibility with the LDAP backend

Yes, absolutely understood and a good point.

In the case of engines that don't support filtering, ordering, or other 
DB-like operations, then a pagination implementation in the controller 
would have to be provided. Not efficient, but better than nothing.

-jay

> Mark.
>
> -----Original Message-----
> From: Jay Pipes [mailto:jaypipes at gmail.com]
> Sent: Tuesday, August 13, 2013 10:10 AM
> To: openstack-dev at lists.openstack.org
> Subject: Re: [openstack-dev] [keystone] Pagination
>
> On 08/13/2013 12:55 PM, Lyle, David (Cloud Services) wrote:
>> The marker/limit pagination scheme is inferior.
>
> A bold statement that flies in the face of experience and the work already done in all the other projects.
>
>   >The use of page/page_size allows access to arbitrary pages, whereas limit/marker only allows forward progress.
>
> I don't see this as a particularly compelling use case considering the performance manifestations of using LIMIT OFFSET pagination.
>
>   >In Horizon's use case, with page/page_size we can provide the user access to any page they have already visited, rather than just the previous page (using prev/next links returned in the response).
>
> I don't see this as a particularly useful thing, but in any case, you could still do this by keeping the markers for previous pages on the client (Horizon) side.
>
> The point of marker/limit is to eliminate poor performance of LIMIT OFFSET queries and to force proper index usage in the listing queries.
>
> You can see the original discussion about this from more than two years and even see where I was originally arguing for a LIMIT OFFSET strategy and was brought around to the current limit/marker strategy by the responses of Justin Santa Barbara and Greg Holt:
>
> https://lists.launchpad.net/openstack/msg02548.html
>
> Best,
> -jay
>
>> -David
>>
>> On 08/13/2013 10:29 AM, Pipes, Jay wrote:
>>
>>> On 08/13/2013 03:05 AM, Yee, Guang wrote:
>>>> Passing the query parameters, whatever they are, into the driver if
>>>> the given driver supports pagination and allowing the driver to
>>>> override the manager default pagination functionality seem reasonable to me.
>>
>>> Please do use the standards that are supported in other OpenStack services already: limit, marker, sort_key and sort_dir.
>>
>>> Pagination is meaningless without a sort key and direction, so picking a sensible default for user/project records is good. I'd go with either created_at (what Glance/Nova/Cinder use..) or with the user/project >UUID.
>>
>>> The Glance DB API pagination is well-documented and clean [1]. I highly recommend it as a starting point.
>>
>>> Nova uses the same marker/limit/sort_key/sort_dir options for queries
>>> that it allows pagination on. An example is the
>>> instance_get_all_by_filters() call [2].
>>
>>> Cinder uses the same marker/limit/sort_key/sort_dir options for query
>>> pagination as well. [3]
>>
>>> Finally, I'd consider supporting the standard change-since parameter for listing operations. Both Nova [4] and Glance [5] support the parameter, which is useful for tools that poll the APIs for "new" >events/records.
>>
>>> In short, go with what is already a standard in the other projects...
>>
>>> Best,
>>> -jay
>>
>>> [1]
>>> https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/
>>> api.py#L429
>>> [2]
>>> https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.
>>> py#L1709
>>> [3]
>>> https://github.com/openstack/cinder/blob/master/cinder/common/sqlalch
>>> emyutils.py#L33
>>> [4]
>>> https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.
>>> py#L1766
>>> [5]
>>> https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/
>>> api.py#L618
>>
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>