Open Stack

Followup note: Though briefly mentioned by John, I like to emphasize this also affects COPY (or PUT with X-Copy-From) requests, and #1 (upping the lb timeout) is really the only solution unless we go crazy and implement async requests with status checks. Well, another weird solution is to have Swift return useless response bodies very slowly as a keep alive. :)


On Aug 10, 2012, at 11:31 AM, John Dickinson <me at not.mn> wrote:

> In a standard swift deployment, the proxy server is running behind a load balancer and/or an SSL terminator. At SwiftStack, we discovered an issue that may arise from some config parameters in this layer, and we'd like to share it with other swift deployers.
> 
> Symptom:
> 
> Users updating metadata (ie POST) on larger objects get 503 error responses. However, there are no error responses logged by swift.
> 
> Cause:
> 
> Since POSTs are implemented, by default, as a server-side copy in swift and there is no traffic between the user and swift during the server-side copy, the LB or SSL terminator times out before the operation is done.
> 
> Solution:
> 
> Two options:
> 
> 1) Raise the timeout in the LB/SSL terminator config. For example, with pound change the "TimeOut" for the swift backend. pound defaults to 15 seconds. The appropriate value is however log it takes to do a server side copy of your largest object. If you have a 1gbps network, it will take about 160 seconds to copy a 5GB object ((8*5*2**30)/((2**30)/4) -- the divide by 4 is because the 1gbps link is used to read one stream (the original) and write the new copy (3 replicas)).
> 
> 2) Change the behavior of POSTs to not do a server-side copy. This will make POSTs faster, but it will prevent all metadata values from being updated (notably, Content-Type will not be able to be modified with a POST). Also, this will not make the issue go away with user-initiated server-side copies.
> 
> I would recommend the first solution, unless your workload makes heavy use of POSTs.
> 
> Hoep this helps.
> 
> --John
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp