[Openstack] [Swift] Database index on object(deleted, name)

Clay Gerrard clay.gerrard at gmail.com
Wed Mar 11 07:19:12 UTC 2015


On Tue, Mar 10, 2015 at 10:30 PM, Shrinand Javadekar <
shrinand at maginatics.com> wrote:

> Has there been any analysis done of the cost incurred by
> this index?


No formal analysis beyond some benchmarking back when gholt, redbo and
wreese were trying to figure out what to do about crappy performance of
listings (> 1000ms) on this one 4GB container with "only" a few thousand
objects but millions of rows because it was being used as a work queue.  If
you're offering to dig into it and publish some quantification I don't
think you'd be duplicating any currently available material on the subject
- knock yourself out.


> 1. Having this index will slow down writes into the object table. But
> by how much?
> 2. The index will use some disk space. How much?
>

Yeah that's the thing.  It's not free, sure - but it's definitely cheaper
than the name index - and w/o the name index I don't think prefix queries
would work at all once you get a few hundred thousand objects in a
container - much less a million or more.  Not sure what you hope to achieve
- always better to know than not know?

OTOH, if you're really interested in the guts of swift's sqlite datastores
it might be more interesting/useful to see if there's any room to help with
container sharding [1] - I'm sure matt wouldn't mind a review on the spec,
even if it's just typos and clarifications.  Either that or try again to
see if sqlite's WAL auto checkpointing [2] actually works [3] on recent
versions and improves throughput on container updates over the current
.pending & batch insert trick [4].

1. https://blueprints.launchpad.net/swift/+spec/container-sharding
2. https://www.sqlite.org/wal.html#ckpt
3.
https://github.com/openstack/swift/commit/ed69db162aa9daa84cc3c86aec0af0c3f840acf6
4.
https://github.com/openstack/swift/blob/master/swift/container/backend.py#L697

Good luck!

-Clay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150311/66462465/attachment.html>


More information about the Openstack mailing list