<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi,<br>
I suffer the same issue that our swift system gets slower and
slower. I tried almost all the guide in the document but it does
little help.<br>
<br>
Maybe swift is not suitable to the circumstance of putting and
getting many little files. Maybe is it more suitable to the large
object such as the image of OS? I am not quite sure.<br>
<br>
Best Regards<br>
Jonathan Lu<br>
<br>
On 2013/8/5 18:35, Hua ZZ Zhang wrote:<br>
</div>
<blockquote
cite="mid:OFF92BE715.B26527CA-ON48257BBE.00387DCE-48257BBE.003A4679@cn.ibm.com"
type="cite">
<p><font face="sans-serif" size="2">Here's the deployment guide
which can give you options for tuning container replicator and
auditor, such as </font><font face="Lucida Sans Unicode"
size="1">concurrency</font><font face="serif" size="3">.</font><br>
<a moz-do-not-send="true"
href="http://docs.openstack.org/developer/swift/deployment_guide.html"><font
face="serif" size="3" color="#0000FF"><u>http://docs.openstack.org/developer/swift/deployment_guide.html</u></font></a><font
face="serif" size="3"> </font><br>
<font face="serif" size="3">hope this can help you.</font><br>
<font face="sans-serif" size="2"><br>
</font><font face="serif" size="1"><b>Best Regards, </b></font><font
face="serif" size="1"><br>
<br>
</font></p>
<hr size="2" width="100%" align="left">
<table border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr valign="top">
<td valign="middle" width="298">
<ul style="padding-left: 0pt">
<font face="serif" size="1"><b>Edward Zhang(张华)</b></font><font
face="serif" size="1"><br>
Advisory Software Engineer<br>
Software Standards & Open Source Software<br>
Emerging Technology Institute(ETI)<br>
IBM China Software Development Lab<br>
e-mail: <a class="moz-txt-link-abbreviated" href="mailto:zhuadl@cn.ibm.com">zhuadl@cn.ibm.com</a><br>
</font>
</ul>
</td>
<td valign="middle" width="298"><img
src="cid:part2.09030400.03080500@gmail.com" alt=""
height="1" width="1" border="0"></td>
</tr>
<tr valign="top">
<td valign="middle" width="298">
<ul style="padding-left: 0pt">
<img src="cid:part3.05060203.00080402@gmail.com"
height="100" width="100">
</ul>
</td>
<td valign="middle" width="298"><img
src="cid:part2.09030400.03080500@gmail.com" alt=""
height="1" width="1" border="0"></td>
</tr>
<tr valign="top">
<td valign="middle" width="298">
<ul style="padding-left: 0pt">
<font face="serif" size="1"><br>
</font>
</ul>
</td>
<td valign="middle" width="298"><img
src="cid:part2.09030400.03080500@gmail.com" alt=""
height="1" width="1" border="0"></td>
</tr>
</tbody>
</table>
<br>
<img src="cid:part6.03070500.09020208@gmail.com" alt="Inactive
hide details for Klaus Schürmann ---08/05/2013 06:13:43
PM---Klaus Schürmann <klaus.schuermann@mediabeam.com>"
height="16" width="16" border="0"><font face="sans-serif"
size="2" color="#424282">Klaus Schürmann ---08/05/2013 06:13:43
PM---Klaus Schürmann <a class="moz-txt-link-rfc2396E" href="mailto:klaus.schuermann@mediabeam.com"><klaus.schuermann@mediabeam.com></a></font><br>
<br>
<table width="100%" border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr valign="top">
<td
style="background-image:url(cid:4__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com);
background-repeat: no-repeat; " width="40%">
<ul style="padding-left: 72pt">
<font face="sans-serif" size="1"><b>Klaus Schürmann
<a class="moz-txt-link-rfc2396E" href="mailto:klaus.schuermann@mediabeam.com"><klaus.schuermann@mediabeam.com></a></b></font><font
face="sans-serif" size="1"> </font>
<p><font face="sans-serif" size="1">08/05/2013 06:14 PM</font></p>
</ul>
</td>
<td width="60%">
<table width="100%" border="0" cellpadding="0"
cellspacing="0">
<tbody>
<tr valign="top">
<td width="1%"><img
src="cid:part2.09030400.03080500@gmail.com"
alt="" height="1" width="58" border="0"><br>
<div align="right"><font face="sans-serif"
size="1">To</font></div>
</td>
<td width="100%"><img
src="cid:part2.09030400.03080500@gmail.com"
alt="" height="1" width="1" border="0"><br>
<ul style="padding-left: 7pt">
<font face="sans-serif" size="1">Hua ZZ
Zhang/China/IBM@IBMCN, </font>
</ul>
</td>
</tr>
<tr valign="top">
<td width="1%"><img
src="cid:part2.09030400.03080500@gmail.com"
alt="" height="1" width="58" border="0"><br>
<div align="right"><font face="sans-serif"
size="1">cc</font></div>
</td>
<td width="100%"><img
src="cid:part2.09030400.03080500@gmail.com"
alt="" height="1" width="1" border="0"><br>
<ul style="padding-left: 7pt">
<font face="sans-serif" size="1"><a class="moz-txt-link-rfc2396E" href="mailto:openstack@lists.openstack.org">"openstack@lists.openstack.org"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:openstack@lists.openstack.org"><openstack@lists.openstack.org></a></font>
</ul>
</td>
</tr>
<tr valign="top">
<td width="1%"><img
src="cid:part2.09030400.03080500@gmail.com"
alt="" height="1" width="58" border="0"><br>
<div align="right"><font face="sans-serif"
size="1">Subject</font></div>
</td>
<td width="100%"><img
src="cid:part2.09030400.03080500@gmail.com"
alt="" height="1" width="1" border="0"><br>
<ul style="padding-left: 7pt">
<font face="sans-serif" size="1">AW: [Openstack]
[SWIFT] PUTs and GETs getting slower</font>
</ul>
</td>
</tr>
</tbody>
</table>
<table border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr valign="top">
<td width="58"><img
src="cid:part2.09030400.03080500@gmail.com"
alt="" height="1" width="1" border="0"></td>
<td width="336"><img
src="cid:part2.09030400.03080500@gmail.com"
alt="" height="1" width="1" border="0"></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<br>
<font face="Calibri" size="2" color="#1F497D">Here is my
container-server.conf:</font><br>
<font face="Calibri" size="2" color="#1F497D"> </font><br>
<font face="Calibri" size="2" color="#1F497D">[DEFAULT]</font><br>
<font face="Calibri" size="2" color="#1F497D">log_name =
container-server</font><br>
<font face="Calibri" size="2" color="#1F497D">log_facility =
LOG_LOCAL6</font><br>
<font face="Calibri" size="2" color="#1F497D">log_level = INFO</font><br>
<font face="Calibri" size="2" color="#1F497D">log_address =
/dev/log</font><br>
<font face="Calibri" size="2" color="#1F497D">bind_ip = 10.12.1.1</font><br>
<font face="Calibri" size="2" color="#1F497D">workers = 8</font><br>
<font face="Calibri" size="2" color="#1F497D">log_statsd_host =
10.11.2.1</font><br>
<font face="Calibri" size="2" color="#1F497D">log_statsd_port =
8125</font><br>
<font face="Calibri" size="2" color="#1F497D">log_statsd_default_sample_rate
= 0.25</font><br>
<font face="Calibri" size="2" color="#1F497D">log_statsd_metric_prefix
= Storage01</font><br>
<font face="Calibri" size="2" color="#1F497D"> </font><br>
<font face="Calibri" size="2" color="#1F497D">[pipeline:main]</font><br>
<font face="Calibri" size="2" color="#1F497D">pipeline = recon
container-server</font><br>
<font face="Calibri" size="2" color="#1F497D"> </font><br>
<font face="Calibri" size="2" color="#1F497D">[filter:recon]</font><br>
<font face="Calibri" size="2" color="#1F497D">use =
egg:swift#recon</font><br>
<font face="Calibri" size="2" color="#1F497D">recon_cache_path =
/var/cache/swift</font><br>
<font face="Calibri" size="2" color="#1F497D"> </font><br>
<font face="Calibri" size="2" color="#1F497D">[app:container-server]</font><br>
<font face="Calibri" size="2" color="#1F497D">use =
egg:swift#container</font><br>
<font face="Calibri" size="2" color="#1F497D"> </font><br>
<font face="Calibri" size="2" color="#1F497D">[container-replicator]</font><br>
<font face="Calibri" size="2" color="#1F497D">concurrency = 2</font><br>
<font face="Calibri" size="2" color="#1F497D"> </font><br>
<font face="Calibri" size="2" color="#1F497D">[container-updater]</font><br>
<font face="Calibri" size="2" color="#1F497D">concurrency = 1</font><br>
<font face="Calibri" size="2" color="#1F497D"> </font><br>
<font face="Calibri" size="2" color="#1F497D">[container-auditor]</font><br>
<font face="Calibri" size="2" color="#1F497D"> </font><br>
<font face="Calibri" size="2" color="#1F497D">[container-sync]</font><br>
<font face="Calibri" size="2" color="#1F497D"> </font><br>
<font face="Calibri" size="2" color="#1F497D"> </font><br>
<font face="Calibri" size="2" color="#1F497D">--------</font>
<p><font face="Arial" size="2">For the second question, Swift
doesn't use </font><font face="Courier New" size="2"> updatedb.mlocate</font><font
face="Arial" size="2"> in my mind. it should be safe to
disable it.<br>
Based on the TOP output, it seems that the CPU workload of
container replicator and auditor are too high.<br>
Perhaps you need to tune the config file to decrease their
workload. paste your config file of<br>
container servers.<br>
</font><font face="Times New Roman" size="1"><b><br>
Best Regards, </b></font><br>
</p>
<hr size="2" width="100%" align="left">
<table class="MsoNormalTable" border="0" cellpadding="0"
cellspacing="0">
<tbody>
<tr valign="top">
<td valign="middle" width="298">
<ul style="padding-left: 36pt">
<font face="Times New Roman" size="1"><b>Edward Zhang(</b></font><font
face="PMingLiU" size="1"><b>张华</b></font><font
face="Times New Roman" size="1"><b>)</b></font><font
face="Times New Roman" size="1"><br>
Advisory Software Engineer<br>
Software Standards & Open Source Software<br>
Emerging Technology Institute(ETI)<br>
IBM China Software Development Lab<br>
e-mail: </font><a moz-do-not-send="true"
href="mailto:zhuadl@cn.ibm.com"><font face="Times New
Roman" size="1" color="#0000FF"><u>zhuadl@cn.ibm.com</u></font></a>
</ul>
</td>
<td valign="middle" width="298"><img
src="cid:part16.06060700.09060608@gmail.com" height="1"
width="1"></td>
</tr>
<tr valign="top">
<td valign="middle" width="298">
<ul style="padding-left: 36pt">
<img src="cid:part3.05060203.00080402@gmail.com"
height="100" width="100">
</ul>
</td>
<td valign="middle" width="298"><img
src="cid:part16.06060700.09060608@gmail.com" height="1"
width="1"></td>
</tr>
<tr valign="top">
<td valign="middle" width="298"><img
src="cid:part2.09030400.03080500@gmail.com" alt=""
height="1" width="1" border="0"></td>
<td valign="middle" width="298"><img
src="cid:part16.06060700.09060608@gmail.com" height="1"
width="1"></td>
</tr>
</tbody>
</table>
<font face="Times New Roman" size="3"><br>
</font><img src="cid:part6.03070500.09020208@gmail.com"
alt="Inactive hide details for Klaus Schürmann ---08/05/2013
05:43:08 PM---Klaus Schürmann
<klaus.schuermann@mediabeam.com>" height="16" width="16"><font
face="Arial" size="2" color="#424282">Klaus Schürmann
---08/05/2013 05:43:08 PM---Klaus Schürmann <</font><a
moz-do-not-send="true"
href="mailto:klaus.schuermann@mediabeam.com"><font face="Arial"
size="2" color="#0000FF"><u>klaus.schuermann@mediabeam.com</u></font></a><font
face="Arial" size="2" color="#424282">></font>
<p>
<table class="MsoNormalTable" width="100%" border="0"
cellpadding="0" cellspacing="0">
<tbody>
<tr valign="top">
<td width="46%">
<ul style="padding-left: 36pt">
<font face="Arial" size="1"><b>Klaus Schürmann <</b></font><a
moz-do-not-send="true"
href="mailto:klaus.schuermann@mediabeam.com"><font
face="Arial" size="1" color="#0000FF"><b><u>klaus.schuermann@mediabeam.com</u></b></font></a><font
face="Arial" size="1"><b>></b></font><font
face="Arial" size="1"> </font><font face="Times New
Roman" size="3"> </font>
<p><font face="Arial" size="1">08/05/2013 05:26 PM</font></p>
</ul>
</td>
<td width="54%">
<table class="MsoNormalTable" width="100%" border="0"
cellpadding="0" cellspacing="0">
<tbody>
<tr valign="top">
<td width="13%"><img
src="cid:part24.00020402.09000704@gmail.com"
height="1" width="58">
<div align="right"><font face="Arial" size="1">To</font></div>
</td>
<td width="87%"><img
src="cid:part16.06060700.09060608@gmail.com"
height="1" width="1">
<ul style="padding-left: 36pt">
<font face="Arial" size="1">"</font><a
moz-do-not-send="true"
href="mailto:openstack@lists.openstack.org"><font
face="Arial" size="1" color="#0000FF"><u>openstack@lists.openstack.org</u></font></a><font
face="Arial" size="1">" <</font><a
moz-do-not-send="true"
href="mailto:openstack@lists.openstack.org"><font
face="Arial" size="1" color="#0000FF"><u>openstack@lists.openstack.org</u></font></a><font
face="Arial" size="1">>, </font>
</ul>
</td>
</tr>
<tr valign="top">
<td width="13%"><img
src="cid:part24.00020402.09000704@gmail.com"
height="1" width="58">
<div align="right"><font face="Arial" size="1">cc</font></div>
</td>
<td width="87%"><img
src="cid:part16.06060700.09060608@gmail.com"
height="1" width="1"></td>
</tr>
<tr valign="top">
<td width="13%"><img
src="cid:part24.00020402.09000704@gmail.com"
height="1" width="58">
<div align="right"><font face="Arial" size="1">Subject</font></div>
</td>
<td width="87%"><img
src="cid:part16.06060700.09060608@gmail.com"
height="1" width="1">
<ul style="padding-left: 36pt">
<font face="Arial" size="1">[Openstack]
[SWIFT] PUTs and GETs getting slower</font>
</ul>
</td>
</tr>
</tbody>
</table>
<font face="Times New Roman" size="3"> </font>
<p><br>
<table class="MsoNormalTable" border="0"
cellpadding="0" cellspacing="0">
<tbody>
<tr valign="top">
<td width="58"><img
src="cid:part16.06060700.09060608@gmail.com"
height="1" width="1"></td>
<td width="336"><img
src="cid:part16.06060700.09060608@gmail.com"
height="1" width="1"></td>
</tr>
</tbody>
</table>
</p>
</td>
</tr>
</tbody>
</table>
<font face="Courier New" size="2"><br>
Hi,<br>
<br>
my Swift cluster is working for three weeks now as a mail
storage and the performance is getting slower and slower.<br>
<br>
This is my hardware configuration:<br>
<br>
- 1 hardware-loadbalancer<br>
- 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU<br>
- 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x
2TB HDD SATA 6GB/s<br>
<br>
The cluster contains:<br>
<br>
279.000 containers<br>
8.170.000 objects = 984 GByte<br>
<br>
I have about 3.000.000 GETs per day and 1.500.000 PUTs.<br>
<br>
The load is raising up to 12 now:<br>
<br>
top - 11:06:39 up 114 days, 23:10, 1 user, load average:
10.88, 11.26, 11.22<br>
Tasks: 175 total, 3 running, 172 sleeping, 0 stopped, 0
zombie<br>
%Cpu(s): 15.6 us, 9.8 sy, 0.0 ni, 69.3 id, 4.5 wa, 0.0 hi,
0.8 si, 0.0 st<br>
KiB Mem: 16386408 total, 14829360 used, 1557048 free,
30796 buffers<br>
KiB Swap: 17575932 total, 132604 used, 17443328 free,
5572360 cached<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND<br>
25880 swift 20 0 81400 15m 3352 D 34.5 0.1 2838:36
/usr/bin/python /usr/bin/swift-container-replicator
/etc/swift/container-server.conf<br>
24282 swift 20 0 81084 14m 3244 S 19.2 0.1 3835:23
/usr/bin/python /usr/bin/swift-container-auditor
/etc/swift/container-server.conf<br>
29816 swift 20 0 1523m 27m 2032 D 9.6 0.2 1132:58
/usr/bin/python /usr/bin/swift-object-server
/etc/swift/object-server.conf<br>
29812 swift 20 0 1523m 27m 2032 R 7.0 0.2 1169:50
/usr/bin/python /usr/bin/swift-object-server
/etc/swift/object-server.conf<br>
29815 swift 20 0 1523m 27m 2032 D 6.6 0.2 1132:45
/usr/bin/python /usr/bin/swift-object-server
/etc/swift/object-server.conf<br>
29809 swift 20 0 1523m 26m 2032 D 5.6 0.2 1163:53
/usr/bin/python /usr/bin/swift-object-server
/etc/swift/object-server.conf<br>
30684 swift 20 0 84496 16m 2468 D 5.6 0.1 1959:57
/usr/bin/python /usr/bin/swift-container-server
/etc/swift/container-server.conf<br>
13853 snmp 20 0 51844 3356 1800 S 5.3 0.0 32:14.90
/usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p
/var/run/snmpd.pid<br>
29810 swift 20 0 1523m 27m 2032 D 5.3 0.2 1132:09
/usr/bin/python /usr/bin/swift-object-server
/etc/swift/object-server.conf<br>
29813 swift 20 0 1525m 27m 2032 D 5.3 0.2 1177:23
/usr/bin/python /usr/bin/swift-object-server
/etc/swift/object-server.conf<br>
29811 swift 20 0 1524m 28m 2032 D 5.0 0.2 1169:37
/usr/bin/python /usr/bin/swift-object-server
/etc/swift/object-server.conf<br>
30677 swift 20 0 84932 17m 2468 D 5.0 0.1 1992:49
/usr/bin/python /usr/bin/swift-container-server
/etc/swift/container-server.conf<br>
29814 swift 20 0 1524m 27m 2032 D 4.6 0.2 1181:11
/usr/bin/python /usr/bin/swift-object-server
/etc/swift/object-server.conf<br>
30681 swift 20 0 84980 17m 2468 S 4.3 0.1 1999:12
/usr/bin/python /usr/bin/swift-container-server
/etc/swift/container-server.conf<br>
30678 swift 20 0 84684 17m 2468 D 4.0 0.1 1973:14
/usr/bin/python /usr/bin/swift-container-server
/etc/swift/container-server.conf<br>
30682 swift 20 0 84224 16m 2468 S 4.0 0.1 1960:55
/usr/bin/python /usr/bin/swift-container-server
/etc/swift/container-server.conf<br>
30685 swift 20 0 84356 16m 2468 D 3.6 0.1 1979:40
/usr/bin/python /usr/bin/swift-container-server
/etc/swift/container-server.conf<br>
7086 root 20 0 0 0 0 S 3.3 0.0 0:00.78
[kworker/6:1]<br>
30683 swift 20 0 84160 16m 2468 S 3.3 0.1 1981:38
/usr/bin/python /usr/bin/swift-container-server
/etc/swift/container-server.conf<br>
30679 swift 20 0 83852 16m 2468 S 3.0 0.1 1965:53
/usr/bin/python /usr/bin/swift-container-server
/etc/swift/container-server.conf<br>
5935 root 20 0 0 0 0 S 2.7 0.0 0:16.38
[kworker/2:1]<br>
6832 root 20 0 0 0 0 S 2.7 0.0 0:02.78
[kworker/4:0]<br>
7153 swift 20 0 80300 9.8m 2040 D 2.7 0.1 0:00.08
/usr/bin/python /usr/bin/swift-container-updater
/etc/swift/container-server.conf<br>
1840 root 20 0 0 0 0 S 2.3 0.0 0:59.87
[kworker/0:1]<br>
9630 swift 20 0 75796 6272 1708 D 2.0 0.0 1889:16
/usr/bin/python /usr/bin/swift-object-auditor
/etc/swift/object-server.conf<br>
9656 swift 20 0 87424 17m 2304 S 2.0 0.1 748:29.97
/usr/bin/python /usr/bin/swift-account-server
/etc/swift/account-server.conf<br>
9659 swift 20 0 86908 17m 2304 S 2.0 0.1 803:09.64
/usr/bin/python /usr/bin/swift-account-server
/etc/swift/account-server.conf<br>
9653 swift 20 0 85816 16m 2304 S 1.7 0.1 723:34.42
/usr/bin/python /usr/bin/swift-account-server
/etc/swift/account-server.conf<br>
9654 swift 20 0 87500 17m 2304 R 1.7 0.1 769:23.97
/usr/bin/python /usr/bin/swift-account-server
/etc/swift/account-server.conf<br>
9655 swift 20 0 85956 16m 2408 S 1.7 0.1 783:28.18
/usr/bin/python /usr/bin/swift-account-server
/etc/swift/account-server.conf<br>
9658 swift 20 0 87808 18m 2304 S 1.7 0.1 767:31.82
/usr/bin/python /usr/bin/swift-account-server
/etc/swift/account-server.conf<br>
32616 root 20 0 0 0 0 S 1.7 0.0 0:18.13
[kworker/3:0]<br>
1826 syslog 20 0 243m 5124 856 S 1.3 0.0 329:10.12
rsyslogd -c5<br>
6944 root 20 0 0 0 0 S 1.3 0.0 0:01.08
[kworker/7:2]<br>
9652 swift 20 0 86896 17m 2412 S 1.3 0.1 663:32.21
/usr/bin/python /usr/bin/swift-account-server
/etc/swift/account-server.conf<br>
9657 swift 20 0 86016 16m 2408 S 1.3 0.1 706:22.08
/usr/bin/python /usr/bin/swift-account-server
/etc/swift/account-server.conf<br>
51 root 20 0 0 0 0 S 1.0 0.0 134:26.37
[kswapd0]<br>
4548 root 20 0 0 0 0 S 1.0 0.0 0:11.69
[kworker/1:2]<br>
5362 root 20 0 0 0 0 S 1.0 0.0 0:10.95
[kworker/5:3]<br>
3 root 20 0 0 0 0 S 0.7 0.0 604:49.60
[ksoftirqd/0]<br>
52 root 20 0 0 0 0 S 0.7 0.0 68:32.58
[kswapd1]<br>
7412 swift 20 0 80300 9704 2780 S 0.7 0.1 566:00.53
/usr/bin/python /usr/bin/swift-container-updater
/etc/swift/container-server.conf<br>
9580 swift 20 0 75632 7140 2728 D 0.7 0.0 2218:06
/usr/bin/python /usr/bin/swift-object-auditor
/etc/swift/object-server.conf<br>
22178 root 20 0 7404 992 700 D 0.7 0.0 20:28.93
/usr/bin/updatedb.mlocate<br>
<br>
Is this a memory problem or a harddisk problem?<br>
<br>
And second question: On the server the mlocate database is
rising. Can I disable the updatedb.mlocate process and empty
the database?<br>
<br>
Thanks<br>
Klaus<br>
<br>
<br>
<br>
_______________________________________________<br>
Mailing list: </font><a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><font
face="Courier New" size="2" color="#0000FF"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</u></font></a><font
face="Courier New" size="2"><br>
Post to : </font><a moz-do-not-send="true"
href="mailto:openstack@lists.openstack.org"><font
face="Courier New" size="2" color="#0000FF"><u>openstack@lists.openstack.org</u></font></a><font
face="Courier New" size="2"><br>
Unsubscribe : </font><a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><font
face="Courier New" size="2" color="#0000FF"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</u></font></a><font
face="Courier New" size="2"><br>
</font><br>
<br>
</p>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Mailing list: <a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
Post to : <a class="moz-txt-link-abbreviated" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>
Unsubscribe : <a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
</pre>
</blockquote>
<br>
</body>
</html>