<div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote">On Tue, Aug 13, 2013 at 3:10 AM, Henry Nash <span dir="ltr"><<a href="mailto:henryn@linux.vnet.ibm.com" target="_blank">henryn@linux.vnet.ibm.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word">Hi<div><br></div><div>So few comebacks to the various comments:</div>
<div><br></div><div>1) While I understand the idea that a client would follow the next/prev links returned in collections, I wasn't aware that we considered 'page'/'per-page' as not standardized. We list these explicitly throughout the identity API spec (look in each List 'entity' example).</div>
</div></blockquote><div><br></div><div>They were essentially relics from a very early draft of the spec that were thoughtlessly copy/pasted around (I'm guilty of this myself)... they were recently cleaned up and removed from the spec.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>How I imagined it would work would be:</div>
<div><br></div><div>a) If a client did not include 'page' in the url we would not paginate</div></div></blockquote><div><br></div><div>Make that a deployment option? per_page could simply default to a very high value.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>b) Once we are paginating, a client can either build the next/prevs urls themselves if they want (by incrementing/decrementing the page number), or just follow the next/prev links (which come with the appropriate 'page=x' in them) returned in the collection which saves them having to do this.</div>
</div></blockquote><div><br></div><div>I'm obviously very opposed to this because it unreasonably forces a single approach to pagination across all drivers.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word"><div>c) Regarding implementation, the controller would continue to be able to paginate on behalf of drivers that couldn't, but those paginate-aware drivers would take over that capability (and indicate this to the controller the state of the pagination so that it can build the correct next/prev links)</div>
<div><br></div><div>2) On the subject of huge enumerates, options are:</div><div>a) Support a backend manager scoped (i.e. identity/assignent/token) limit in the conf file which would be honored by drivers. Assuming that you set this larger than your pagination limit, this would make sense whether your driver is paginating or not in terms of minimizing the delay in responding data as well as not messing up pagination. In the non-paginated case when we hit the limit, should we indicate this to the client? Maybe a 206 return code? Although i) not quite sure that meets http standards, and ii) would we break a bunch of clients by doing this?</div>
</div></blockquote><div><br></div><div>I'm not clear on what kind of limit you're referring to? A 206 sounds unexpected for this use case though.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word"><div>b) We scrap the whole idea of pagination, and just set a conf limit as in 2a). To make this work of course, we must implement any defined filters in the backend (otherwise we still end up with today's performance problems - remember that today, in general, filtering is done in the controller on a full enumeration of the entities in question). I was planning to implement this backend filtering anyway as part of (or on top of) my change, since we are holding (at least one of) our hands behind our backs right now by not doing so. And our filters need to be powerful, do we support wildcards for example, e.g. GET /users?name = fred* ?</div>
</div></blockquote><div><br></div><div>There were some discussions on this topic from about a year ago that I'd love to continue. I don't want to invent a new "language," but we do need to settle on an approach that we can apply across a wide variety of backends. That probably means keeping it very simple (like your example). Asterisks need to be URL encoded, though. One suggestion I particularly liked (which happens to avoid claiming perfectly valid characters - asterisks - as special characters) was to adopt the syntax used in the django ORM's filter function:</div>
<div><br></div><div> ?name__startswith=Fred<br></div><div> ?name__istartswith=fred<br></div><div> ?name__endswith=Fred<br></div><div> ?name__iendswith=fred<br></div><div> ?name__contains=Fred</div> ?name__icontains=fred</div>
<div class="gmail_quote"><br></div><div class="gmail_quote">This probably represents the immediately useful subset of parameters for us, but for more:</div><div class="gmail_quote"><br></div><div class="gmail_quote"> <a href="https://docs.djangoproject.com/en/dev/topics/db/queries/">https://docs.djangoproject.com/en/dev/topics/db/queries/</a><br>
</div><div class="gmail_quote"><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word"><span class=""><font color="#888888"><div> </div><div>Henry</div></font></span><div><div class="h5"><div><br></div><div><div><div>On 13 Aug 2013, at 04:40, Adam Young wrote:</div><br><blockquote type="cite">
<div text="#000000" bgcolor="#FFFFFF">
<div>On 08/12/2013 09:22 PM, Miller, Mark M
(EB SW Cloud - R&D - Corvallis) wrote:<br>
</div>
<blockquote type="cite">
<div><p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The
main reason I use user lists (i.e. keystone user-list) is to
get the list of usernames/IDs for other keystone commands. I
do not see the value of showing all of the users in an LDAP
server when they are not part of the keystone database (i.e.
do not have roles assigned to them). Performing a “keystone
user-list” command against the HP Enterprise Directory locks
up keystone for about 1 ˝ hours in that it will not perform
any other commands until it is done. If it is decided that
user lists are necessary, then at a minimum they need to be
paged to return control back to keystone for another
command.</span></p>
</div>
</blockquote>
<br>
We need a way to tell HP ED to limit the number of rows, and to do
filtering.<br>
<br>
We have a bug for the second part. I'll open one for the limit.<br>
<br>
<blockquote type="cite">
<div><p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p><p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Mark<u></u><u></u></span></p><p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-style:solid none none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0in 0in"><p class=""><b><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext">
Adam Young [<a href="mailto:ayoung@redhat.com" target="_blank">mailto:ayoung@redhat.com</a>]
<br>
<b>Sent:</b> Monday, August 12, 2013 5:27 PM<br>
<b>To:</b> <a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>
<b>Subject:</b> Re: [openstack-dev] [keystone]
Pagination<u></u><u></u></span></p>
</div>
</div><p class=""><u></u> <u></u></p>
<div><p class="">On 08/12/2013 05:34 PM, Henry Nash wrote:<u></u><u></u></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt"><p class="">Hi <u></u><u></u></p>
<div><p class=""><u></u> <u></u></p>
</div>
<div><p class="">I'm working on extending the pagination
into the backends. Right now, we handle the pagination in
the v3 controller class....and in fact it is disabled
right now and we return the whole list irrespective of
whether page/per-page is set in the query string, e.g.:<u></u><u></u></p>
</div>
</blockquote><p class="">Pagination is a broken concept. We should
not be returning lists so long that we need to paginate.
Instead, we should have query limits, and filters to refine
the queries.<br>
<br>
Some people are doing full user lists against LDAP. I don't
need to tell you how broken that is. Why do we allow
user-list at the Domain (or unscoped level)?
<br>
<br>
I'd argue that we should drop enumeration of objects in
general, and certainly limit the number of results that come
back. Pagination in LDAP requires cursors, and thus continuos
connections from Keystone to LDAP...this is not a scalable
solution.<br>
<br>
Do we really need this?<br>
<br>
<br>
<br>
<u></u><u></u></p>
<div><p class=""><u></u> <u></u></p>
</div>
<div>
<div><p class=""><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">
</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(6,63,244)">def</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">
<b>paginate</b>(cls, context, refs):<u></u><u></u></span></p>
</div>
<div><p class=""><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">
</span><i><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(0,179,57)">"""Paginates
a list of references by page & per_page query
strings."""</span></i><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(0,179,57)"><u></u><u></u></span></p>
</div>
<div><p class=""><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">
</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(203,203,203)">#
FIXME(<u>dolph</u>): client needs to support pagination
first<u></u><u></u></span></p>
</div>
<div><p class=""><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">
</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(6,63,244)">return</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">
refs<u></u><u></u></span></p>
</div>
<div><p class=""><span style="font-size:7.5pt;font-family:Helvetica,sans-serif"><u></u> <u></u></span></p>
</div>
<div><p class=""><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">
page = context[</span><i><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(0,179,57)">'query_string'</span></i><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">].get(</span><i><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(0,179,57)">'page'</span></i><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">,
</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(149,15,11)">1</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">)<u></u><u></u></span></p>
</div>
<div><p class=""><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">
per_page = context[</span><i><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(0,179,57)">'query_string'</span></i><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">].get(</span><i><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(0,179,57)">'per_page'</span></i><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">,
</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(149,15,11)">30</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">)<u></u><u></u></span></p>
</div>
<div><p class=""><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">
</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(6,63,244)">return</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">
refs[per_page * (page -
</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif;color:rgb(149,15,11)">1</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif">):per_page
* page]<u></u><u></u></span></p>
</div>
</div>
<div><p class=""><span style="font-size:7.5pt;font-family:Helvetica,sans-serif"><u></u> <u></u></span></p>
</div>
<div><p class=""><span style="font-family:Helvetica,sans-serif">I
wonder both for the V3 controller (which still needs to
handle pagination for backends that do not support it) and
the backends that do....whether we could use wether 'page'
is defined in the query-string as an indicator as to
whether we should paginate or not? That way clients who
can handle it can ask for it, those that don'twill just
get everything. </span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif"><u></u><u></u></span></p>
</div>
<div><p class=""><span style="font-size:7.5pt;font-family:Helvetica,sans-serif"><u></u> <u></u></span></p>
</div>
<div><p class=""><span style="font-family:Helvetica,sans-serif">Henry</span><span style="font-size:7.5pt;font-family:Helvetica,sans-serif"><u></u><u></u></span></p>
</div>
<div><p class=""><u></u> <u></u></p>
</div><p class=""><br>
<br>
<br>
<u></u><u></u></p>
<pre>_______________________________________________<u></u><u></u></pre>
<pre>OpenStack-dev mailing list<u></u><u></u></pre>
<pre><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><u></u><u></u></pre>
<pre><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><u></u><u></u></pre><p class=""><u></u> <u></u></p>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br></div></div></div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><br></div>-Dolph
</div></div>