Open Stack

On Mon, Dec 19, 2016 at 11:54 AM, Monty Taylor <mordred at inaugust.com> wrote:

> Hey all,
>
> In the docs for listing containers:
>
> http://developer.openstack.org/api-ref/object-storage/?
> expanded=show-account-details-and-list-containers-detail
>
> it says that limit is optional. However, if you give it
> full_listing=True, swiftclient always sends at least a second query with
> a marker set to the name of the last element of the original return
> value. This leads me believe there is a default value for limit server
> side that swiftclient is trying to account for.
>
> Looking through capabilities, it seems there is:
>
> swift.account_listing_limit
> swift.container_listing_limit
>
>
Yeah there is a server side limit, and as you surmise you can find it when
you query the cluster for the information (swift capabilities or  curl -i
http://<swift_endpoint>/info |python -m json.tool).

The swift client will look at capabilities and be smart about it.

We limit the size of the response to the server size limit. full_listing is
swiftclient trying to be helpful and paginate for you.


> I'm guessing swift.account_listing_limit would apply to getting the list
> of containers, and container_listing_limit would apply to listing
> objects in a container, yeah?
>

Exactly, yes.


>
> Relatedly, there is a X-Account-Container-Count header. Testing against
> rackspace public shows me that it contains the total count of containers
> even if I provide a limit parameter:
>
> >>> r=c.get('/?format=json&limit=2')
> >>> r.headers['X-Account-Container-Count']
> '4'
>
> Would that also be the case if I had hit the
> swift.account_listing_limit limit?


Yup, the container-count (account metadata) and object-count (container
metadata) are how many are in the account/container NOT in the response. As
it's gathering the metadata information from the account/container. If you
fire off a HEAD (or stat in swiftclient) you'll get the same counts.




> I ask because it seems like it would be ever-so-mildly more efficient to
> look at the returned count and the total count and only do the loop if
> there is a mismatch - but I'm curious if there is something I should
> account for I'm not seeing. (I'm working on making direct REST calls,
> but starting with swiftclient's behavior to make sure I'm doing things
> right)
>
> Also - the question about limit behavior when not given seems like it
> could be added to the docs - and I'd be happy to make that patch if I'm
> reading things right.
>

Looking at the API docs briefly, this could be more clearly stated.
Especially as the first paragraph in the Account GET mentions if you get
'limit' back then there are more objects, and the first paragraph of the
Container GET mentions there being a default limit on the server side if
the parameters are omitted:
http://developer.openstack.org/api-ref/object-storage/?expanded=show-account-details-and-list-containers-detail,show-container-details-and-list-objects-detail

So yeah could definitely be cleaned up.


> Thanks!
> Monty
>
> No thank you :)

> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


I was talking to Richard from horizon while they've been building a new
Swift UI, and for displaying many objects, they've mentioned liking someway
of knowing if there are any more results, so they can continue to paginate
as people scroll etc, our discussions have gone further then just knowing
if there's more now. And so I plan on having a play with helping them
paginate better on the Swift side for Horizon.

At the moment, if you get limit results back (where limit is what you
define or defaults to the server default) then there is a good chance there
are more objects. Just keeping track of the total can be a little
problematic as It's possible for that count to grow or shrink between
requests. So if you hit limit, then there is probably more and do another
request is easiest.

Regards,
Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161219/6a6c2aad/attachment.html>