Open Stack

Hi,all

David said: users will simply try to get rid of their volumes ALL at the same time and this is putting a lot of pressure on the SAN servicing those volumes and since the hardware isn't replying fast enough, the process then fall in D state and are waiting for IOs to complete which slows down everything.
The system must tolerate this kind of behavior. The status of process "dd" will fall in D state with the pressure of  SAN.

In my opinion, we should rethink the way of wiping the data in the volumes. Filling in the device with /dev/zero with "dd" command was the most primitive method.  The standard scsi command "write same" could be taken into considered.

Once the LBA was provided and the command was sent to the SAN , the storage device(SAN) could write the same-data into the LUN or volumes. The "dd" operation  can be offloaded to the storage array to execute.

Thanks,

Qi


Reference:

1)       http://manpages.ubuntu.com/manpages/karmic/man8/sg_write_same.8.html

2)       http://storagegaga.wordpress.com/2012/01/06/why-vaai/


________________________________
Qi Xiaozhen
CLOUD OS PDU, IT Product Line, Huawei Enterprise Business Group
Mobile: +86 13609283376    Tel: +86 29-89191578
Email: qixiaozhen at huawei.com <mailto:qixiaozhen at huawei.com>


From: David Hill [mailto:david.hill at ubisoft.com]
Sent: Saturday, November 02, 2013 6:21 AM
To: openstack at lists.openstack.org
Subject: [Openstack] Wiping of old cinder volumes

Hi guys,

                I was wondering there was some better way of wiping the content of an old EBS volume before actually deleting the logical volume in cinder ?  Or perhaps, configure or add the possibility to configure the number of parallel "dd" processes that will be spawn at the same time...
Sometimes, users will simply try to get rid of their volumes ALL at the same time and this is putting a lot of pressure on the SAN servicing those volumes and since the hardware isn't replying fast enough, the process then fall in D state and are waiting for IOs to complete which slows down everything.
Since this process isn't (in my opinion) as critical as a EBS write or read, perhaps we should be able to throttle the speed of disk wiping or number of parallel wipings to something that wouldn't affect the other read/write that are most probably more critical.

Here is a small capture of the processes :
cinder   23782  0.7  0.2 248868 20588 ?        S    Oct24  94:23 /usr/bin/python /usr/bin/cinder-volume --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/volume.log
cinder   23790  0.0  0.5 382264 46864 ?        S    Oct24   9:16  \_ /usr/bin/python /usr/bin/cinder-volume --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/volume.log
root     32672  0.0  0.0 175364  2648 ?        S    21:48   0:00  |   \_ sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--2e86d686--de67--4ee4--992d--72818c70d791 count=102400 bs=1M co
root     32675  0.0  0.1 173636  8672 ?        S    21:48   0:00  |   |   \_ /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--2e86d686--de67--4ee4--992d--72818c70d7
root     32681  3.2  0.0 106208  1728 ?        D    21:48   0:47  |   |       \_ /bin/dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--2e86d686--de67--4ee4--992d--72818c70d791 count=102400 bs=1M conv=fdatasync
root     32674  0.0  0.0 175364  2656 ?        S    21:48   0:00  |   \_ sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--d54a1c96--63ca--45cb--a597--26194d45dcdf count=102400 bs=1M co
root     32676  0.0  0.1 173636  8672 ?        S    21:48   0:00  |   |   \_ /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--d54a1c96--63ca--45cb--a597--26194d45dc
root     32683  3.2  0.0 106208  1724 ?        D    21:48   0:47  |   |       \_ /bin/dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--d54a1c96--63ca--45cb--a597--26194d45dcdf count=102400 bs=1M conv=fdatasync
root     32693  0.0  0.0 175364  2656 ?        S    21:48   0:00  |   \_ sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--048dae36--b225--4266--b21e--af4b66eae6cd count=102400 bs=1M co
root     32694  0.0  0.1 173632  8668 ?        S    21:48   0:00  |   |   \_ /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--048dae36--b225--4266--b21e--af4b66eae6
root     32707  3.2  0.0 106208  1728 ?        D    21:48   0:46  |   |       \_ /bin/dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--048dae36--b225--4266--b21e--af4b66eae6cd count=102400 bs=1M conv=fdatasync
root       342  0.0  0.0 175364  2648 ?        S    21:48   0:00  |   \_ sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--45251e8e--0c54--4e8f--9446--4e92801976ab count=102400 bs=1M co
root       343  0.0  0.1 173636  8672 ?        S    21:48   0:00  |   |   \_ /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--45251e8e--0c54--4e8f--9446--4e92801976
root       347  3.2  0.0 106208  1728 ?        D    21:48   0:45  |   |       \_ /bin/dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--45251e8e--0c54--4e8f--9446--4e92801976ab count=102400 bs=1M conv=fdatasync
root       380  0.0  0.0 175364  2656 ?        S    21:48   0:00  |   \_ sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--1d9dfb31--dc06--43d5--bc1f--93b6623ff8c4 count=102400 bs=1M co
root       382  0.0  0.1 173632  8668 ?        S    21:48   0:00  |   |   \_ /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--1d9dfb31--dc06--43d5--bc1f--93b6623ff8
root       388  3.2  0.0 106208  1724 ?        R    21:48   0:45  |   |       \_ /bin/dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--1d9dfb31--dc06--43d5--bc1f--93b6623ff8c4 count=102400 bs=1M conv=fdatasync
root       381  0.0  0.0 175364  2648 ?        S    21:48   0:00  |   \_ sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--60971d47--d3c5--44ef--9d43--d461c364d148 count=102400 bs=1M co
root       384  0.0  0.1 173636  8672 ?        S    21:48   0:00  |   |   \_ /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--60971d47--d3c5--44ef--9d43--d461c364d1
root       391  3.2  0.0 106208  1728 ?        D    21:48   0:45  |   |       \_ /bin/dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--60971d47--d3c5--44ef--9d43--d461c364d148 count=102400 bs=1M conv=fdatasync
root       383  0.0  0.0 175364  2648 ?        S    21:48   0:00  |   \_ sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--856080db--4f8c--4063--9c47--69acb8460e50 count=102400 bs=1M co
root       386  0.0  0.1 173632  8668 ?        S    21:48   0:00  |   |   \_ /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--856080db--4f8c--4063--9c47--69acb8460e
root       389  3.1  0.0 106208  1724 ?        D    21:48   0:45  |   |       \_ /bin/dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--856080db--4f8c--4063--9c47--69acb8460e50 count=102400 bs=1M conv=fdatasync
root       385  0.0  0.0 175364  2652 ?        S    21:48   0:00  |   \_ sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--f8f98d80--044f--4d4a--983f--d1186556f886 count=102400 bs=1M co
root       387  0.0  0.1 173632  8668 ?        S    21:48   0:00  |   |   \_ /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--f8f98d80--044f--4d4a--983f--d1186556f8
root       392  3.1  0.0 106208  1728 ?        D    21:48   0:45  |   |       \_ /bin/dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--f8f98d80--044f--4d4a--983f--d1186556f886 count=102400 bs=1M conv=fdatasync
root       413  0.0  0.0 175364  2652 ?        S    21:48   0:00  |   \_ sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--0e89696a--492b--494c--81fa--7e834b9f31f4 count=102400 bs=1M co
root       414  0.0  0.1 173636  8672 ?        S    21:48   0:00  |       \_ /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--0e89696a--492b--494c--81fa--7e834b9f31
root       420  3.1  0.0 106208  1728 ?        D    21:48   0:45  |           \_ /bin/dd if=/dev/zero of=/dev/mapper/cinder--volumes-volume--0e89696a--492b--494c--81fa--7e834b9f31f4 count=102400 bs=1M conv=fdatasync
cinder   23791  0.0  0.5 377464 41968 ?        S    Oct24   7:46  \_ /usr/bin/python /usr/bin/cinder-volume --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/volume.log

iostat output :
dm-23             0.00     0.00    0.00 18408.00     0.00    71.91     8.00   503.06   28.83   0.05 100.00
dm-25             0.00     0.00    0.00 20544.00     0.00    80.25     8.00   597.24   30.56   0.05 100.10
dm-29             0.00     0.00    0.00 19232.00     0.00    75.12     8.00   531.80   27.62   0.05 100.10
dm-34             0.00     0.00    0.00 20128.00     0.00    78.62     8.00   498.10   24.92   0.05 100.00
dm-39             0.00     0.00    0.00 18355.00     0.00    71.70     8.00   534.77   28.98   0.05 100.00
dm-59             0.00     0.00    0.00 18387.00     0.00    71.82     8.00   587.79   32.10   0.05 100.00
dm-96             0.00     0.00    0.00 16480.00     0.00    64.38     8.00   467.96   27.51   0.06 100.00
dm-97             0.00     0.00    0.00 17024.00     0.00    66.50     8.00   502.25   29.21   0.06 100.00
dm-98             0.00     0.00    0.00 20704.00     0.00    80.88     8.00   655.67   31.37   0.05 100.00

parent dm :
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
dm-0            142.00 74394.00  100.00 2812.00     1.00   302.41   213.38   156.74   52.84   0.34 100.00

Thank you very much ,

Dave

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131104/40d0706e/attachment.html>