Open Stack

On Tue, Aug 13, 2013 at 3:10 AM, Henry Nash <henryn at linux.vnet.ibm.com>wrote:

> Hi
>
> So few comebacks to the various comments:
>
> 1) While I understand the idea that a client would follow the next/prev
> links returned in collections, I wasn't aware that we considered
> 'page'/'per-page' as not standardized. We list these explicitly throughout
> the identity API spec (look in each List 'entity' example).
>

They were essentially relics from a very early draft of the spec that were
thoughtlessly copy/pasted around (I'm guilty of this myself)... they were
recently cleaned up and removed from the spec.


> How I imagined it would work would be:
>
> a) If a client did not include 'page' in the url we would not paginate
>

Make that a deployment option? per_page could simply default to a very high
value.


> b) Once we are paginating, a client can either build the next/prevs urls
> themselves if they want (by incrementing/decrementing the page number), or
> just follow the next/prev links (which come with the appropriate 'page=x'
> in them) returned in the collection which saves them having to do this.
>

I'm obviously very opposed to this because it unreasonably forces a single
approach to pagination across all drivers.


> c) Regarding implementation, the controller would continue to be able to
> paginate on behalf of drivers that couldn't, but those paginate-aware
> drivers would take over that capability (and indicate this to the
> controller the state of the pagination so that it can build the correct
> next/prev links)
>
> 2) On the subject of huge enumerates, options are:
> a) Support a backend manager scoped (i.e. identity/assignent/token) limit
> in the conf file which would be honored by drivers.  Assuming that you set
> this larger than your pagination limit, this would make sense whether your
> driver is paginating or not in terms of minimizing the delay in responding
> data as well as not messing up pagination.  In the non-paginated case when
> we hit the limit, should we indicate this to the client?  Maybe a 206
> return code?  Although i) not quite sure that meets http standards, and ii)
> would we break a bunch of clients by doing this?
>

I'm not clear on what kind of limit you're referring to? A 206 sounds
unexpected for this use case though.


> b) We scrap the whole idea of pagination, and just set a conf limit as in
> 2a).  To make this work of course, we must implement any defined filters in
> the backend (otherwise we still end up with today's performance problems -
> remember that today, in general,  filtering is done in the controller on a
> full enumeration of the entities in question).  I was planning to implement
> this backend filtering anyway as part of (or on top of) my change, since we
> are holding (at least one of) our hands behind our backs right now by not
> doing so.  And our filters need to be powerful, do we support wildcards for
> example, e.g. GET /users?name = fred*  ?
>

There were some discussions on this topic from about a year ago that I'd
love to continue. I don't want to invent a new "language," but we do need
to settle on an approach that we can apply across a wide variety of
backends. That probably means keeping it very simple (like your example).
Asterisks need to be URL encoded, though. One suggestion I particularly
liked (which happens to avoid claiming perfectly valid characters -
asterisks - as special characters) was to adopt the syntax used in the
django ORM's filter function:

  ?name__startswith=Fred
  ?name__istartswith=fred
  ?name__endswith=Fred
  ?name__iendswith=fred
  ?name__contains=Fred
  ?name__icontains=fred

This probably represents the immediately useful subset of parameters for
us, but for more:

  https://docs.djangoproject.com/en/dev/topics/db/queries/


> Henry
>
> On 13 Aug 2013, at 04:40, Adam Young wrote:
>
>  On 08/12/2013 09:22 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis)
> wrote:
>
> The main reason I use user lists (i.e. keystone user-list) is to get the
> list of usernames/IDs for other keystone commands. I do not see the value
> of showing all of the users in an LDAP server when they are not part of the
> keystone database (i.e. do not have roles assigned to them). Performing a
> “keystone user-list” command against the HP Enterprise Directory locks up
> keystone for about 1 ½ hours in that it will not perform any other commands
> until it is done.  If it is decided that user lists are necessary, then at
> a minimum they need to be paged to return control back to keystone for
> another command.
>
>
> We need a way to tell HP ED to limit the number of rows, and to do
> filtering.
>
> We have a bug for the second part.  I'll open one for the limit.
>
>  ****
>
> ** **
>
> Mark****
>
> ** **
>
> *From:* Adam Young [mailto:ayoung at redhat.com <ayoung at redhat.com>]
> *Sent:* Monday, August 12, 2013 5:27 PM
> *To:* openstack-dev at lists.openstack.org
> *Subject:* Re: [openstack-dev] [keystone] Pagination****
>
> ** **
>
> On 08/12/2013 05:34 PM, Henry Nash wrote:****
>
> Hi ****
>
> ** **
>
> I'm working on extending the pagination into the backends.  Right now, we
> handle the pagination in the v3 controller class....and in fact it is
> disabled right now and we return the whole list irrespective of whether
> page/per-page is set in the query string, e.g.:****
>
> Pagination is a broken concept. We should not be returning lists so long
> that we need to paginate.  Instead, we should have query limits, and
> filters to refine the queries.
>
> Some people are doing full user lists against LDAP.  I don't need to tell
> you how broken that is.  Why do we allow user-list at the Domain (or
> unscoped level)?
>
> I'd argue that we should drop enumeration of objects in general, and
> certainly limit the number of results that come back.  Pagination in LDAP
> requires cursors, and thus continuos connections from Keystone to
> LDAP...this is not a scalable solution.
>
> Do we really need this?
>
>
>
> ****
>
> ** **
>
>     def *paginate*(cls, context, refs):****
>
>         *"""Paginates a list of references by page & per_page query
> strings."""*****
>
>         # FIXME(*dolph*): client needs to support pagination first****
>
>         return refs****
>
> ** **
>
>         page = context[*'query_string'*].get(*'page'*, 1)****
>
>         per_page = context[*'query_string'*].get(*'per_page'*, 30)****
>
>         return refs[per_page * (page - 1):per_page * page]****
>
> ** **
>
> I wonder both for the V3 controller (which still needs to handle
> pagination for backends that do not support it) and the backends that
> do....whether we could use wether 'page' is defined in the query-string as
> an indicator as to whether we should paginate or not?  That way clients who
> can handle it can ask for it, those that don'twill just get everything.  *
> ***
>
> ** **
>
> Henry****
>
> ** **
>
>
>
>
> ****
>
> _______________________________________________****
>
> OpenStack-dev mailing list****
>
> OpenStack-dev at lists.openstack.org****
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev****
>
> ** **
>
>
> _______________________________________________
> OpenStack-dev mailing listOpenStack-dev at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>  _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

-Dolph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130813/2e8d1d63/attachment.html>