<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Just before the Austin hackathon, we uploaded 5 patches related to the cluster federation blueprint<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://blueprints.launchpad.net/swift/+spec/cluster-federation">https://blueprints.launchpad.net/swift/+spec/cluster-federation</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">one of those patches, <a href="https://review.openstack.org/51242">
https://review.openstack.org/51242</a> dealt with the mechanics of forwarding data from a middleware component to/from a different cluster. This patch had two goals:<o:p></o:p></p>
<p class="MsoNormal">1. Maximize code reuse while minimizing core code change: any new code is simple and doesn’t impact connection management, or the way the core handles data proxying between connections.<o:p></o:p></p>
<p class="MsoNormal">2. Refactor at the level of the ring and introduce the concept of a TargetRing to show that container-forwarding can be seen as a kind of storage policy while, again, minimizing core code change.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have just pushed a new patch, <a href="https://review.openstack.org/55674">
https://review.openstack.org/55674</a> which presents an alternative way to achieve the code reuse objective by creating a swift/common/http_utils module that can be used by both middleware and core proxy-server code (and possibly other code as well such as
backend daemons, container-sync being a possible candidate).<o:p></o:p></p>
<p class="MsoNormal">Like the first patch, it also includes a very basic piece of middleware that forwards all requests to a configured target cluster.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am including more details about the patch further down so if you’re interested, please keep reading.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am interested in finding out which approach people prefer and get feedback on the patch itself.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Eric<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">More details<o:p></o:p></p>
<p class="MsoNormal">--<o:p></o:p></p>
<p class="MsoNormal">Most of the code complexity in http_utils comes from the fact that for object PUT data must be transferred between the client connection and the backends/target connection and, at the moment, this uses decoupling queues and extra threads
to produce to / consume from the queues. It is basically very close to the code from helper methods already in base.py and obj.py so reusing the code from http_utils in proxy-server means that the following code can be removed (only commented out for now):<o:p></o:p></p>
<p class="MsoNormal">base.py: _<i>make_app</i>_iter<o:p></o:p></p>
<p class="MsoNormal">obj.py: _<i>send</i>_file plus a chunk of code embedded in PUT<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Unlike the previous patch, this alternative patch has basic support for ssl based on the httplib.HTTPSConnection class used by bufferedhttp.py. It may form a good basis for more generic use, in particular by the container-sync daemon which
must interact with proxies from other clusters. I understand there is a wish to remove the dependency on swiftclient in this daemon and the code in http_utils could be a good step forward towards that objective.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Both patches rely on a callback from proxy/server.py (just like already exists for calling back the authorization middleware) to give a handle on the controller object that is created anew for each request. The app given to a middleware
at instantiation time is the next filter in the pipeline and only maps to the proxy-server application if the middleware is placed just before proxy-server in the paste-deploy config pipeline. As far as I know, a callback from server.py is the only way a
middleware can have a handle on proxy-server or a controller while being flexible about its placement in the pipeline.<o:p></o:p></p>
<p class="MsoNormal">In the case of the original patch, the handle on the controller is needed to be able to call reused controller methods such as make_requests, etc. In the case of the alternative patch, only a handle on the proxy-server application is needed
for http purposes so that configuration values (timeouts, queue depth, etc.) can be reused. However, in both cases, a fully functional container-forwarding will require a handle on the controller to implement complex multi-container use cases (COPY, SLO,
DLO, versioning) where forwarded containers could reside in different clusters. <o:p>
</o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>