Open Stack

Hi Lincoln,

Glad to see this thread rebooted. Here are some comments on the metadata 
search API proposal. I would be glad to see feedback from others too.

Data Types
The proposal suggests each attribute has one of 3 types: string, numeric, 
date. How do we deduce the type for user metadata attributes i.e. 
object_meta_<name> attributes etc.? This is important for range searches 
since types such as dates are not sorted correctly when represented as 
strings. This seems to warrant a configuration file which contains schema 
information. In addition, using attribute naming conventions could help 
set the default behavior e.g. object_meta_start_date could refer to a date 
by default since it has the suffix _date. Also, additional types which 
seem important are a floating point type and a type for representing 
geo-spatial coordinates and/or general spatial information. Ideally the 
set of types which can be handled should be extensible. 

System Metadata
Regarding system metadata, many attributes are part of this proposal which 
are not currently part of Swift. There should be a minimal set of 
attributes - probably those attributes which are part of Swift - which 
should be available for indexing by all vendor implementations. Other than 
that, different vendors would have different requirements for generating 
new system metadata for indexing, and it probably shouldn't be a mandatory 
part of the API. Also, in general, the user should be able to control 
which user and system metadata attributes are indexed in order to control 
the amount of resources invested in metadata indexing/search.

Let me know what you think.

Best regards
Paula

Paula Ta-Shma, Ph.D.
Storage Research
IBM Research - Haifa
Phone: +972.3.7689402
Email: paula at il.ibm.com




From:   "Thomas, Lincoln (HP Storage R&D)" <Lincoln.Thomas at hp.com>
To:     "openstack-dev at lists.openstack.org" 
<openstack-dev at lists.openstack.org>, 
Date:   11/01/2014 04:09 AM
Subject:        [openstack-dev]  [Swift] Metadata Search API



Rebooting this thread now that I've reorg'd the Wiki page.

The proposed REST API spec for searching system and custom metadata in 
Swift, across accounts, containers, and objects, is now posted at:

https://wiki.openstack.org/wiki/MetadataSearchAPI

I've also made the first modification to the API since the Icehouse design 
summit where I introduced this project: adding "metadata:" per 
conversation with Paula Ta-Shma's team at IBM Storage Research.

The home page for this project remains at:

https://wiki.openstack.org/wiki/MetadataSearch 

See that home page for further details, and the history of this email 
thread. Feel free to edit the Wiki as described on the home page!

As Brian Cline (SoftLayer) said so eloquently in this thread:

> Today, about the best one can do is iterate through everything and 
inspect metadata 
> along the way - obviously an infinitely expensive (and hilariously 
insane) 
> operation.

> If there are any others who have implemented search in Swift, please 
speak up and 
> help shape this. We both want to get community consensus on a standard 
search API, 
> then get a pluggable reference implementation into Swift.

> This is all work-in-progress stuff, but we'd welcome any feedback, 
concerns, 
> literal jumps for joy, etc. in this thread, both on the API and on a 
reference 
> architecture.

Thanks in advance,
Lincoln Thomas (IRC lincolnt)
System/Software Engineer, HP Storage R&D
Portland, OR, USA,  +1 (503) 757-6274


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140123/babc6e70/attachment.html>