[Swift] [kolla] Swift issues in one cluster

Albert Braden ozzzo at yahoo.com
Fri Jun 24 16:46:40 UTC 2022


 AFAIK we are running vanilla Swift. Clients usually connect to the Swift endpoint by running "swift" or "openstack container" commands, and Swift uses exabgp to route the traffic. I'm not exactly sure what the Java library is doing but I can dig more into that if it would help.

I see swift_proxy_server and swift_haproxy containers running on every Swift node. Is that not the normal configuration?

According to [1] it's pretty easy to reproduce:

"You can reproduce this by issuing a GET request for a few hundred MB file and never consuming the response, but keep the client socket open. Swift will log a 499 but the socket does not always close."     On Friday, June 24, 2022, 12:02:05 PM EDT, Clay Gerrard <clay.gerrard at gmail.com> wrote:  
 
 On Fri, Jun 24, 2022 at 10:29 AM Albert Braden <ozzzo at yahoo.com> wrote:
>
> "Having another look at that issue, it sounds like slow client shouldn't be handled by OpenStack services but rather with a load balancer, especially if the service is Internet facing"
>
> I don't understand what is being recommended here.

I think they were suggesting using a http proxy application - maybe
haproxy - will have more options to protect network resources from
misbehaving clients than the swift proxy application.  Like kicking
off keep-alive connections after a while, or slow clients that hang up
resources.

> We have 60 Swift servers, and customer traffic goes directly to those servers. It seems like a load-balancer would be a performance-reducing bottleneck.

That's cool, do you use round robin dns or something?

> Is there any hope of getting this bug fixed?

If we can reproduce the problem you're seeing there's some chance we
could offer a solution through just a code change, but it's going to
be difficult if repro requires haproxy in the pipeline.  If there is a
problem w/o haproxy, it might have more to do with eventlet.wsgi or
python's base http server than swift... can you affirm the issue when
clients talk directly to the python/eventlet/swift application?

-- 
Clay Gerrard

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220624/dd0307a7/attachment-0001.htm>


More information about the openstack-discuss mailing list