Open Stack

Hi All,

We are running Swift 1.4.8 with 8 nodes and 4 zones.

We recently added 4 SSD drives to one each to 4 of our storage nodes. 
The accounts and container rings were then rebalanced to ensure this 
data doesn't sit on spinning disks. Since the rebalance was done, we 
have noticed something unusual in the statistics returned from within 
swift.

This is the command being run to grab the statistics:

swift -v -A https://127.0.0.1:8080/auth/v1.0 -U <USERNAME> -K <PASS> 
stat

Before the changes, the statistics looked like this:

===
Wed, 30 Jul 2014 10:51:26 +0100
Array
(
     [X-Account-Object-Count] => 81473735
     [X-Account-Bytes-Used] => 34156718530011
     [X-Account-Container-Count] => 6510
)
Wed, 30 Jul 2014 10:51:36 +0100
Array
(
     [X-Account-Object-Count] => 81473735
     [X-Account-Bytes-Used] => 34156718530011
     [X-Account-Container-Count] => 6510
)
Wed, 30 Jul 2014 10:51:46 +0100
Array
(
     [X-Account-Object-Count] => 81698252
     [X-Account-Bytes-Used] => 34213134745373
     [X-Account-Container-Count] => 6510
)
Wed, 30 Jul 2014 10:51:56 +0100
Array
(
     [X-Account-Object-Count] => 81687266
     [X-Account-Bytes-Used] => 34209086906883
     [X-Account-Container-Count] => 6510
)
Wed, 30 Jul 2014 10:52:06 +0100
Array
(
     [X-Account-Object-Count] => 81687418
     [X-Account-Bytes-Used] => 34209165517185
     [X-Account-Container-Count] => 6510
)
Wed, 30 Jul 2014 10:52:16 +0100
Array
(
     [X-Account-Object-Count] => 81405109
     [X-Account-Bytes-Used] => 34105818678331
     [X-Account-Container-Count] => 6510
)
Wed, 30 Jul 2014 10:52:26 +0100
Array
(
     [X-Account-Object-Count] => 81460103
     [X-Account-Bytes-Used] => 34127360552723
     [X-Account-Container-Count] => 6510
)
===

Since the rebalancing, the statistics seem to show that 
X-Account-Bytes-Used has dropped by around 7TB and 
X-Account-Object-Count seems to have dropped to somewhere between 60M - 
70M objects. The statistics now seem to jump around wildly, as can be 
seen below.

===
Tue, 05 Aug 2014 12:32:49 +0100
Array
(
     [X-Account-Object-Count] => 59242579
     [X-Account-Bytes-Used] => 24304403925249
     [X-Account-Container-Count] => 6603
)
Tue, 05 Aug 2014 12:32:59 +0100
Array
(
     [X-Account-Object-Count] => 58817476
     [X-Account-Bytes-Used] => 24167437130211
     [X-Account-Container-Count] => 6603
)
Tue, 05 Aug 2014 12:33:09 +0100
Array
(
     [X-Account-Object-Count] => 63760679
     [X-Account-Bytes-Used] => 25828018327577
     [X-Account-Container-Count] => 6603
)
Tue, 05 Aug 2014 12:33:19 +0100
Array
(
     [X-Account-Object-Count] => 66724351
     [X-Account-Bytes-Used] => 27197208718607
     [X-Account-Container-Count] => 6603
)
Tue, 05 Aug 2014 12:33:29 +0100
Array
(
     [X-Account-Object-Count] => 67222017
     [X-Account-Bytes-Used] => 27465314723569
     [X-Account-Container-Count] => 6603
)
Tue, 05 Aug 2014 12:33:39 +0100
Array
(
     [X-Account-Object-Count] => 67214198
     [X-Account-Bytes-Used] => 27536268561101
     [X-Account-Container-Count] => 6603
)
Tue, 05 Aug 2014 12:33:49 +0100
Array
(
     [X-Account-Object-Count] => 68353884
     [X-Account-Bytes-Used] => 28017869874871
     [X-Account-Container-Count] => 6603
)
===

The above is repeated, the count increases, then drops back to down. 
The question I have is, why would this happen? We definitely did not 
delete anything, so as far as I am concerned data was just moved around.

You can see the behaviour on these graphs - 
http://www.preeto.co.uk/SwiftStats.PNG - Note how prior to the change 
(2014-07-31), the totalbytes and totalobjects graphs are fairly static.

Regards,

Pritpal