Open Stack

On 16/03/16 00:51, Peter Brouwer wrote:

> Ah, good info. Followup question, assume worse case ( just to emphasis
> the situation) , one copy ( replication = 1 ) , disk approaching its max
> capacity.
> How can you monitor this situation, i.e. to avoid the disk full scenario
> and
> if the disk is full, what type of error is returned?
>

Let's do an example: 4 storage nodes (obj1...obj4) each with 1 disk 
(vdb) added to ring. Replication set to 1.

Firstly write a 1G object (to see where it is gonna go)...host obj1, 
disk vdb, partition 1003):

obj1 $ ls -l 
/srv/node/vdb/objects/1003/d31/fae796287c852f0833316a3dadfb3d31/
total 1048580
-rw------- 1 swift swift 1073741824 Mar 16 10:15 1458079557.01198.data


Then remove it

obj1 $ ls -l 
/srv/node/vdb/objects/1003/d31/fae796287c852f0833316a3dadfb3d31/
total 4
-rw------- 1 swift swift 0 Mar 16 10:47 1458078463.80396.ts


...and use up space on obj1/vdb (dd a 29G file into /srv/node/vdb somewhere)

obj1 $ df -m|grep vdb
/dev/vdb           30705 29729       977  97% /srv/node/vdb


Add object again (ends up on obj4 instead...handoff node)

obj4 $ ls -l 
/srv/node/vdb/objects/1003/d31/fae796287c852f0833316a3dadfb3d31/
total 1048580
-rw------- 1 swift swift 1073741824 Mar 16 11:06 1458079557.01198.data


So swift is coping with the obj1/vdb disk being too full. Remove again 
and exhaust space on all disks (dd again):

@obj[1-4] $ df -h|grep vdb
/dev/vdb         30G   30G  977M  97% /srv/node/vdb


Now attempt to write 1G object again

swiftclient.exceptions.ClientException:
Object PUT failed:
http://192.168.122.61:8080/v1/AUTH_9a428d5a6f134f829b2a5e4420f512e7/con0/obj0 
503 Service Unavailable


So we get an http 503 to show that the put has failed.


Now re monitoring. Out of the box swift-recon cover this:

proxy1 $ swift-recon -dv
===============================================================================
--> Starting reconnaissance on 4 hosts
===============================================================================
[2016-03-16 13:16:54] Checking disk usage now
-> http://192.168.122.63:6000/recon/diskusage: [{u'device': u'vdc', 
u'avail': 32162807808, u'mounted': True, u'used': 33718272, u'size': 
32196526080}, {u'device': u'vdb', u'avail': 1024225280, u'mounted': 
True, u'used': 31172300800, u'size': 32196526080}]
-> http://192.168.122.64:6000/recon/diskusage: [{u'device': u'vdc', 
u'avail': 32162807808, u'mounted': True, u'used': 33718272, u'size': 
32196526080}, {u'device': u'vdb', u'avail': 1024274432, u'mounted': 
True, u'used': 31172251648, u'size': 32196526080}]
-> http://192.168.122.62:6000/recon/diskusage: [{u'device': u'vdc', 
u'avail': 32162807808, u'mounted': True, u'used': 33718272, u'size': 
32196526080}, {u'device': u'vdb', u'avail': 1024237568, u'mounted': 
True, u'used': 31172288512, u'size': 32196526080}]
-> http://192.168.122.65:6000/recon/diskusage: [{u'device': u'vdc', 
u'avail': 32162807808, u'mounted': True, u'used': 33718272, u'size': 
32196526080}, {u'device': u'vdb', u'avail': 1024221184, u'mounted': 
True, u'used': 31172304896, u'size': 32196526080}]
Distribution Graph:
   0%    4 
*********************************************************************
  96%    4 
*********************************************************************
Disk usage: space used: 124824018944 of 257572208640
Disk usage: space free: 132748189696 of 257572208640
Disk usage: lowest: 0.1%, highest: 96.82%, avg: 48.4617574245%
===============================================================================


So integrating swift-recon into regular monitoring/alerting 
(collectd/nagios or whatever) is one approach (mind you most folk 
already monitor disk usage data... and there is nothing overly special 
about ensuring you don't run of space)!


> BTW, thanks for the patience for sticking with me in this.

No worries - a good question (once I finally understood it).

regards

Mark