Open Stack

Excerpts from John Dickinson's message of 2017-06-16 11:35:39 -0700:
> 
> On 16 Jun 2017, at 10:51, Clint Byrum wrote:
> 
> > This is great work.
> >
> > I'm sure you've already thought of this, but could you explain why
> > you've chosen not to put the small objects in the k/v store as part of
> > the value rather than in secondary large files?
> 
> I don't want to co-opt an answer from Alex, but I do want to point to some of the other background on this LOSF work.
> 
> https://wiki.openstack.org/wiki/Swift/ideas/small_files
> https://wiki.openstack.org/wiki/Swift/ideas/small_files/experimentations
> https://wiki.openstack.org/wiki/Swift/ideas/small_files/implementation
> 

These are great. Thanks for sharing them, I understand a lot more now.

> Look at the second link for some context to your answer, but the summary is "that means writing a file system, and writing a file system is really hard".
> 

I'm not sure we were thinking the same thing.

I was more asking, why not put the content of the object into the k/v
instead of the big_file_id:offset? My thinking was that for smaller
objects, you would just return the data immediately upon reading the k/v,
rather than then needing to go find the big file and read the offset.
However, I'm painfully aware that those directly involved with the problem
have likely thought of this. However, the experiments don't seem to show
that this was attempted. Perhaps I'm zooming too far out to see the real
problem space. You can all tell me to take my spray paint can and stop
staring at the bike shed if this is just too annoying. Seriously.

Of course, one important thing is, what does one consider "small"? Seems
like there's a size where the memory footprint of storing it in the
k/v would be justifiable if reads just returned immediately from k/v
vs. needing to also go get data from a big file on disk. Perhaps that
size is too low to really matter. I was hoping that this had been
considered and there was documentation, but I don't really see it.

Also the "writing your own filesystem" option in experiments seemed
more like a thing to do if you left the k/v stores out entirely.