<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Hello,</div>
<div class="elementToProof">
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof">
We recently upgraded our swift servers to 20.04 from 18.04. After the upgrades were completed we are seeing 1 server where the ssync receiver keeps crashing. The crash is preventing hand offs from being cleanup up in the cluster (50 servers). Looking for some
advice to fix/workaround the issue. We have 20 clusters but are only seeing this issue in 1 cluster. It is strange to me that we only see this ssync receiver error in one server out of the ~1000 we are running.</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof">
OS: Ubuntu 20.04.5</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
Swift version: 2.25.2</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
On the receiver side we are seeing the following trace:</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
object-server: senderIp/d1/2166 EXCEPTION in ssync.Receiver: </div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
Traceback (most recent ca<span style="">ll last):</span></div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<span style=""> File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 209, in _chunked_read</span></div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<span style=""> self.chunk_length = int(rfile.readline().split(b";",</span><span style="">1)[0], 16)</span></div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<div class="ContentPasted1 elementToProof">ValueError: invalid literal for int() with base 16: b''</div>
<div class="ContentPasted1 elementToProof"><br>
</div>
<div class="ContentPasted1 elementToProof">During handling of the above exception, another exception occurred:</div>
<div class="ContentPasted1 elementToProof"><br>
</div>
<div class="ContentPasted1 elementToProof">Traceb<span style="">ack (most recent call last):</span></div>
<div class="ContentPasted1 elementToProof"><span style=""> File "/usr/lib/python3/dist-packages/swift/obj/ssync_receiver.py", line 166, in __call__</span></div>
<div class="ContentPasted1 elementToProof"><span style=""> for data in self.missing_che</span><span style="">ck():</span></div>
<div class="ContentPasted1 elementToProof"><span style=""> File "/usr/lib/python3/dist-packages/swift/obj/ssync_receiver.py", line 340, in missing_check</span></div>
<div class="ContentPasted1 elementToProof"><span style=""> line = self.fp.readline(self.app.network_chunk</span><span style="">_size)</span></div>
<div class="ContentPasted1 elementToProof"> File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 226, in readline</div>
<div class="ContentPasted1 elementToProof"> return self._chunked_read(self.rfile, size, True)</div>
<div class="ContentPasted1 elementToProof"> File "<span style="">/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 211, in _chunked_read</span></div>
<div class="ContentPasted1 elementToProof"><span style=""> raise ChunkReadError(err)</span></div>
<div class="ContentPasted1 elementToProof"><span style="">eventlet.wsgi.ChunkReadError: invalid literal</span><span style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt;"> for int() with base 16: b''</span></div>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
On the sender side we are seeing this message when I try to run the reconstructor manually with debug logging:</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2 ContentPasted3 ContentPasted4">
root@use1-saas-p6-paco-9:~# swift-object-reconstructor object-server.conf -p 27999 -v -o
<div class="ContentPasted4">object-reconstructor: Starting 271723</div>
<div class="ContentPasted4">object-reconstructor: Spawned worker 271755 with {'override_partitions': [27999], 'override_devices': ['d10', 'd17', 'd19', 'd9', 'd29', 'd15', 'd2', 'd24', '</div>
<div class="ContentPasted4">d32', 'd1', 'd8', 'd14', 'd11', 'd28', 'd27', 'd6', 'd7', 'd16', 'd13', 'd25', 'd26', 'd20', 'd18', 'd12', 'd4', 'd0', 'd22', 'd3', 'd5', 'd21', 'd33', 'd23',</div>
<div class="ContentPasted4"> 'd30', 'd34', 'd31'], 'multiprocess_worker_index': 0}</div>
<div class="ContentPasted4">object-reconstructor: [worker 1/1 pid=271755] Running object reconstructor in script mode.</div>
<div class="ContentPasted4">object-reconstructor: [worker 1/1 pid=271755] Run listdir on /srv/node/d22/objects-4/27999</div>
<div class="ContentPasted4 elementToProof">object-reconstructor: [worker 1/1 pid=271755] recieverIP:6200/d31/27999 10.0 seconds: connect receive</div>
<div class="ContentPasted4">object-reconstructor: [worker 1/1 pid=271755] 1/26820 (0.00%) partitions reconstructed in 10.14s (0.10/sec, 75h remaining)</div>
<div class="ContentPasted4">object-reconstructor: [worker 1/1 pid=271755] Object reconstruction complete (once). (0.17 minutes)</div>
<div class="ContentPasted4">object-reconstructor: Forked worker 271755 finished</div>
<div class="ContentPasted4">object-reconstructor: Worker 271755 exited</div>
<div class="ContentPasted4">object-reconstructor: Finished 271723</div>
<div class="ContentPasted4">object-reconstructor: Exited 271723</div>
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
Thanks!</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<br>
</div>
<div id="Signature">
<div><strong style="background: rgb(255, 255, 255); margin: 0px; outline: 0px; font-weight: 700; font-family: SimpplrFont, -apple-system, BlinkMacSystemFont, "Segoe UI", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", Arial, sans-serif; font-size: 17px; text-align: start; color: rgb(51, 51, 51);">Reid
Guyett</strong><br style="font-family: SimpplrFont, -apple-system, BlinkMacSystemFont, "Segoe UI", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", Arial, sans-serif; font-size: 17px; text-align: start; color: rgb(51, 51, 51); background-color: rgb(255, 255, 255);">
<span style="font-family: SimpplrFont, -apple-system, BlinkMacSystemFont, "Segoe UI", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", Arial, sans-serif; font-size: 17px; text-align: start; display: inline !important; color: rgb(51, 51, 51); background-color: rgb(255, 255, 255);">Staff
Storage Engineer</span><br style="font-family: SimpplrFont, -apple-system, BlinkMacSystemFont, "Segoe UI", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", Arial, sans-serif; font-size: 17px; text-align: start; color: rgb(51, 51, 51); background-color: rgb(255, 255, 255);">
<span style="font-family: SimpplrFont, -apple-system, BlinkMacSystemFont, "Segoe UI", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", Arial, sans-serif; font-size: 17px; text-align: start; display: inline !important; color: rgb(51, 51, 51); background-color: rgb(255, 255, 255);">Datto,
A Kaseya Company</span><br style="font-family: SimpplrFont, -apple-system, BlinkMacSystemFont, "Segoe UI", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", Arial, sans-serif; font-size: 17px; text-align: start; color: rgb(51, 51, 51); background-color: rgb(255, 255, 255);">
<a href="http://www.datto.com/" target="_blank" style="background: rgb(255, 255, 255); margin: 0px; outline: 0px; user-select: auto; overflow-wrap: break-word; font-family: SimpplrFont, -apple-system, BlinkMacSystemFont, "Segoe UI", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", Arial, sans-serif; font-size: 17px; text-align: start;"><span style="background: transparent; margin: 0px; outline: 0px; color: rgb(11, 83, 148);">www.datto.com</span></a></div>
</div>
</div>
<p style="font-size:8pt; line-height:10pt; font-family: 'Verdana','Verdana',serif; color: gray">
<b>Important Notice:</b> This email is intended to be received only by persons entitled to receive the confidential and legally privileged information it presumptively contains, and this notice constitutes identification as such. Any reading, disclosure, copying,
distribution or use of this information by or to someone who is not the intended recipient, is prohibited. If you received this email in error, please notify us immediately at legal@kaseya.com, and then delete it. To opt-out of receiving emails
<a href="https://info.kaseya.com/email-subscription-center.html">Please click here</a>. The term 'this e-mail' includes any and all attachments.</p>
</body>
</html>