From sorrison@gmail.com Thu Jun 27 10:11:32 2024 From: Sam Morrison To: openstack-discuss@lists.openstack.org Subject: [swift] Weird log corruption or something more serious? Date: Thu, 27 Jun 2024 20:11:05 +1000 Message-ID: <9150116E-4F2B-4149-AAB4-89A0E58F23FB@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============9208716227580048612==" --===============9208716227580048612== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, We are debugging a weird issue with our swift cluster (currently running 2.29= .2) We got a report from a user that they were getting upload failures and then s= ent us this from their logs: 2024-06-25 23:30:06,829 | swiftclient.service | ERROR | Object PUT failed: ht= tps://swift-host/v1/PROJECT_ID_A/prod_result/195e5e8e-334a-11ef-ba70-0a580a64= 0a0b/eval/Dismo-TPR-TNR_Hieracium.pilosella_rangebag.png 401 Unauthorized [fi= rst 60 chars of response] b'

Unauthorized

This server could n= ot verify t' (txn: txab25fcfcf3354bbbaf105-00667b52fe) Searching our logs for that transaction ID I found: 203.101.227.241 127.0.0.1 25/Jun/2024/23/30/06 PUT /v1/AUTH_PROJECT_ID_B/nexu= s-migration/nexus-data/blobs/default/content/vol-27/chap-25/6bc30bae-4f5e-40a= 8-bc63-096679af1a58.bytes HTTP/1.0 401 - rclone/v1.59.0%2Cpython-swiftclient-= 4.2.0 gAAAAABme1L7tlfE... - 131 - txab25fcfcf3354bbbaf105-00667b52fe - 0.0440= - - 1719358206.765954018 1719358206.809994936 0 Note this is a completely different project ID from our side, timestamp is co= rrect too. We also have this:=20 ERROR WSGI: code 400, message Bad request syntax ('\x89PNG') (txn: txab25fcfc= f3354bbbaf105-00667b52fe) (client_ip: XX.XX.XX.XX) Upon further analysis we see this quite a lot, mostly from attempted hackers = but lots from real users and lots seem to be related to rclone. Eg.=20 Jun 27 08:24:46 swift-host proxy-server[70043]: ERROR WSGI: code 400, messag= e Bad request syntax ('PUT /v1/AUTH_PROJECT_ID_B/nexus-migration/nexus-data/e= lasticsearch/nexus/nodes/0/indices/42b6a6a76dae683a647546ac02a6883108ac7df5/0= /index/segments_4PUT /v1/AUTH_PROJECT_ID_B/nexus-migration/nexus-data/elastic= search/nexus/nodes/0/indices/4db62ff1327dfe236d85b1dd1a6a9e8139e7e4cf/0/index= /_67t.cfs HTTP/1.1') (txn: txd57a3d00bb1942c6a390e-00667c952a) (client_ip: XX= .XX.XX.XX) Anyone know what is going on here? Thanks, Sam --===============9208716227580048612==--