Open Stack

It would be my honor and pleasure to continue serving you as Swift PTL.

The world continues to generate and retain ever-increasing amounts of 
data, and Swift continues to rise to the challenge of storing that data 
durably and making it highly available. I'm excited to help us engineer 
now to be ready for next year's data-center demands. To that end, there 
are several projects we'll be advancing; in general, these are not 
short-term, though some pieces will surely come in the next cycle.

As hard-drive sizes increase, the lots-of-small-files problem has become 
a lots-of-files problem. Between 16TB hard drives hitting the market and 
cost concerns pushing us toward ever more drives per box, even 
historically "reasonable" object sizes will produce noticeable memory 
pressure that we'd like to reduce. OVH's explorations in this space give 
us all a leg up that we should take advantage of.

As clusters grow, we must ensure Swift can scale with them. Some aspects 
of that will be relatively straight-forward, like allowing rings to 
support more than 64k devices. Other challenges are less well-defined; 
there are likely improvements that could be made in replication and 
backend protocols, for example, but there is no single way forward. 
There will likely be some experiments that never land on master -- but 
as long as they are in the open and we can all learn from them, they 
will not be failures.

All of that must be driven by what we learn by operating real clusters 
at scale. To do that, we must improve our metrics and monitoring, and 
find ways to observe the system dynamically. Post-facto log analysis is 
not tenable when dealing with tens of thousands of requests per second 
across hundreds of nodes.

At the same time, we cannot neglect our client ecosystem. This extends 
not just to python-swiftclient, but to S3 clients as well. For better or 
worse, S3 has become the de facto standard interface for object storage, 
and we must ensure our compatibility is as seamless as possible. This 
cycle we added S3-compatible versioning, but there is so much more we 
could be doing, from object life-cycle management to bucket inventories. 
As always, we will listen to our users and prioritize addressing their 
pain points.

Tim Burke


-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------