<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi<div><br></div><div>So few comebacks to the various comments:</div><div><br></div><div>1) While I understand the idea that a client would follow the next/prev links returned in collections, I wasn't aware that we considered 'page'/'per-page' as not standardized. We list these explicitly throughout the identity API spec (look in each List 'entity' example). How I imagined it would work would be:</div><div><br></div><div>a) If a client did not include 'page' in the url we would not paginate</div><div>b) Once we are paginating, a client can either build the next/prevs urls themselves if they want (by incrementing/decrementing the page number), or just follow the next/prev links (which come with the appropriate 'page=x' in them) returned in the collection which saves them having to do this.</div><div>c) Regarding implementation, the controller would continue to be able to paginate on behalf of drivers that couldn't, but those paginate-aware drivers would take over that capability (and indicate this to the controller the state of the pagination so that it can build the correct next/prev links)</div><div><br></div><div>2) On the subject of huge enumerates, options are:</div><div>a) Support a backend manager scoped (i.e. identity/assignent/token) limit in the conf file which would be honored by drivers. Assuming that you set this larger than your pagination limit, this would make sense whether your driver is paginating or not in terms of minimizing the delay in responding data as well as not messing up pagination. In the non-paginated case when we hit the limit, should we indicate this to the client? Maybe a 206 return code? Although i) not quite sure that meets http standards, and ii) would we break a bunch of clients by doing this?</div><div>b) We scrap the whole idea of pagination, and just set a conf limit as in 2a). To make this work of course, we must implement any defined filters in the backend (otherwise we still end up with today's performance problems - remember that today, in general, filtering is done in the controller on a full enumeration of the entities in question). I was planning to implement this backend filtering anyway as part of (or on top of) my change, since we are holding (at least one of) our hands behind our backs right now by not doing so. And our filters need to be powerful, do we support wildcards for example, e.g. GET /users?name = fred* ?</div><div> </div><div>Henry</div><div><br></div><div><div><div>On 13 Aug 2013, at 04:40, Adam Young wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 08/12/2013 09:22 PM, Miller, Mark M
(EB SW Cloud - R&D - Corvallis) wrote:<br>
</div>
<blockquote cite="mid:D6182642CE6D2D4FBFCDF99946E249883B361D4E@G9W0343.americas.hpqcorp.net" type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The
main reason I use user lists (i.e. keystone user-list) is to
get the list of usernames/IDs for other keystone commands. I
do not see the value of showing all of the users in an LDAP
server when they are not part of the keystone database (i.e.
do not have roles assigned to them). Performing a “keystone
user-list” command against the HP Enterprise Directory locks
up keystone for about 1 ˝ hours in that it will not perform
any other commands until it is done. If it is decided that
user lists are necessary, then at a minimum they need to be
paged to return control back to keystone for another
command.</span></p>
</div>
</blockquote>
<br>
We need a way to tell HP ED to limit the number of rows, and to do
filtering.<br>
<br>
We have a bug for the second part. I'll open one for the limit.<br>
<br>
<blockquote cite="mid:D6182642CE6D2D4FBFCDF99946E249883B361D4E@G9W0343.americas.hpqcorp.net" type="cite">
<div class="WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Mark<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
Adam Young [<a class="moz-txt-link-freetext" href="mailto:ayoung@redhat.com">mailto:ayoung@redhat.com</a>]
<br>
<b>Sent:</b> Monday, August 12, 2013 5:27 PM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><br>
<b>Subject:</b> Re: [openstack-dev] [keystone]
Pagination<o:p></o:p></span></p>
</div>
</div><p class="MsoNormal"><o:p> </o:p></p>
<div><p class="MsoNormal">On 08/12/2013 05:34 PM, Henry Nash wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"><p class="MsoNormal">Hi <o:p></o:p></p>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div><p class="MsoNormal">I'm working on extending the pagination
into the backends. Right now, we handle the pagination in
the v3 controller class....and in fact it is disabled
right now and we return the whole list irrespective of
whether page/per-page is set in the query string, e.g.:<o:p></o:p></p>
</div>
</blockquote><p class="MsoNormal">Pagination is a broken concept. We should
not be returning lists so long that we need to paginate.
Instead, we should have query limits, and filters to refine
the queries.<br>
<br>
Some people are doing full user lists against LDAP. I don't
need to tell you how broken that is. Why do we allow
user-list at the Domain (or unscoped level)?
<br>
<br>
I'd argue that we should drop enumeration of objects in
general, and certainly limit the number of results that come
back. Pagination in LDAP requires cursors, and thus continuos
connections from Keystone to LDAP...this is not a scalable
solution.<br>
<br>
Do we really need this?<br>
<br>
<br>
<br>
<o:p></o:p></p>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">
</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#063FF4">def</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">
<b>paginate</b>(cls, context, refs):<o:p></o:p></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">
</span><i><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#00B339">"""Paginates
a list of references by page & per_page query
strings."""</span></i><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#00B339"><o:p></o:p></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">
</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#CBCBCB">#
FIXME(<u>dolph</u>): client needs to support pagination
first<o:p></o:p></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">
</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#063FF4">return</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">
refs<o:p></o:p></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">
page = context[</span><i><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#00B339">'query_string'</span></i><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">].get(</span><i><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#00B339">'page'</span></i><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">,
</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#950F0B">1</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">)<o:p></o:p></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">
per_page = context[</span><i><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#00B339">'query_string'</span></i><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">].get(</span><i><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#00B339">'per_page'</span></i><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">,
</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#950F0B">30</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">)<o:p></o:p></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">
</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#063FF4">return</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">
refs[per_page * (page -
</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif";color:#950F0B">1</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif"">):per_page
* page]<o:p></o:p></span></p>
</div>
</div>
<div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div><p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">I
wonder both for the V3 controller (which still needs to
handle pagination for backends that do not support it) and
the backends that do....whether we could use wether 'page'
is defined in the query-string as an indicator as to
whether we should paginate or not? That way clients who
can handle it can ask for it, those that don'twill just
get everything. </span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif""><o:p></o:p></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div><p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Henry</span><span style="font-size:7.5pt;font-family:"Helvetica","sans-serif""><o:p></o:p></span></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div><p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>OpenStack-dev mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></pre><p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br></blockquote></div><br></div></body></html>