<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.bbscopedstyle7263624439947307
{mso-style-name:bbscopedstyle7263624439947307;}
span.EmailStyle18
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">></span></b><span style="font-size:11.0pt;font-family:Calibri"> I realize that you guys run private cloud, so you don’t have to worry about bad actors getting a server from you and doing
malicious things with it. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">We might be a bit more forgiving but we are not as trusting as you might expect ;)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">> But do you have any concerns about the recent research [1] that uses Rowhammer + ksm + transparent hugepages + kvm to change the memory of collocated VM’s?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">>The research showed that they were able to successfully target memory inside other VM’s to do things like modify authorized_keys in memory in such a way that they could successfully login
with their own key<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Yes, the flip-feng-shui.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">We ran a pretty big rowhammer cluster doing both normal and double_sided_rowhammer at our hardware but we did not notice any bitflips.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">So I am still not sure how this actually affects server hardware with DDR4 and ECC.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">From what I could find on the web regarding vulnerable DDR4 (
<a href="http://www.thirdio.com/rowhammer.pdf">http://www.thirdio.com/rowhammer.pdf</a> ) it looks like they tested consumer oriented memory chips.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">e.g. not DDR4 with ECC chips.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Tests that were done with ECC bases memory were all based on DDR3 (which we do not have).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">However, with the expansion of the memory config we will also look if memory is still the limiting factor and revisit KSM based on the usage.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Not just because of rowhammer but also some other KSM things. e.g. KSM does not support hugepages.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Regarding detection: you can look with perf at cache misses at the kvm process but not getting false-positives is hard.<b><o:p></o:p></b></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Robert<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">"Van Leeuwen, Robert" <rovanleeuwen@ebay.com><br>
<b>Date: </b>Friday, November 11, 2016 at 12:10 AM<br>
<b>To: </b>"Kris G. Lindgren" <klindgren@godaddy.com>, Edmund Rhudy <erhudy@bloomberg.net>, "warren@wangspeed.com" <warren@wangspeed.com><br>
<b>Cc: </b>"openstack-operators@lists.openstack.org" <openstack-operators@lists.openstack.org><br>
<b>Subject: </b>Re: [Openstack-operators] Managing quota for Nova local storage?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Thx for your stories,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">I think we are now all doing pretty much the same thing to get around the issue but it still looks like a very useful feature.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">So to share what we (eBay-ECG) are doing:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">We also started out with scaling the flavor disksize to either memory or cpu. (so e.g. large disk == large memory)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">But our users started asking for flavors with quite different specs.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Not being able to give those would be hugely inefficient.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">So now we started giving flavors to specific tenants instead of making them public (and let the quota’s sort it out)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">e.g. a flavor with 8 cores, 12G and 1TB of local storage will only be available for the tenants that really need it.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Looking at our hypervisor stats we either run out of memory or disk before cpu cycles so not having a tunable on disk is inconvenient.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Our latest spec hypervisors have 768GB and we run KSM so we will probably run out of DISK first there.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">We run SSD-only on local storage so that space in the flavor is real $$$.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><br>
We started to run on zfs with compression on our latest config/iteration and that seems to alleviate the pain a bit.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">It is a bit early to tell exactly but it seems to run stable and the compression factor will be around 2.0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">P.S. I noticed my search for blueprints was not good enough so I closed mine and subscribed to the one that’s was already there:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><a href="https://blueprints.launchpad.net/nova/+spec/nova-disk-quota-tracking">https://blueprints.launchpad.net/nova/+spec/nova-disk-quota-tracking</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Robert van Leeuwen</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">"Kris G. Lindgren" <klindgren@godaddy.com><br>
<b>Date: </b>Thursday, November 10, 2016 at 5:18 PM<br>
<b>To: </b>Edmund Rhudy <erhudy@bloomberg.net>, "warren@wangspeed.com" <warren@wangspeed.com>, Robert Van Leeuwen <rovanleeuwen@ebay.com><br>
<b>Cc: </b>"openstack-operators@lists.openstack.org" <openstack-operators@lists.openstack.org><br>
<b>Subject: </b>Re: [Openstack-operators] Managing quota for Nova local storage?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">This is what we have done as well.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">We made our flavors stackable, starting with our average deployed flavor size and making things a multiple of that. IE if our average deployed flavor size is 8GB 120GB of disk, our larger
flavors are multiple of that. So if 16GB 240GB of disk is the average, the next flavor up maybe: 32GB 480GB of disk. From there its easy to then say with 256GB of ram we will average: ~30 VM’s which means we need to have ~3.6TB of local storage per node.
Assuming that you don’t over allocate disk or ram. In practice though you can get a running average of the amount of disk space consumed and work towards that plus a bit of a buffer and run with a disk oversubscription.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">We currently have no desire to remove local storage. We want the root disks to be on local storage. That being said in the future we will most likely give smaller root disks and if people
need more space ask them to provisioning a rbd volume through cinder.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri;color:black">___________________________________________________________________</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri;color:black">Kris Lindgren</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri;color:black">Senior Linux Systems Engineer</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri;color:black">GoDaddy</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">"Edmund Rhudy (BLOOMBERG/ 120 PARK)" <erhudy@bloomberg.net><br>
<b>Reply-To: </b>Edmund Rhudy <erhudy@bloomberg.net><br>
<b>Date: </b>Thursday, November 10, 2016 at 8:47 AM<br>
<b>To: </b>"warren@wangspeed.com" <warren@wangspeed.com>, "rovanleeuwen@ebay.com" <rovanleeuwen@ebay.com><br>
<b>Cc: </b>"openstack-operators@lists.openstack.org" <openstack-operators@lists.openstack.org><br>
<b>Subject: </b>Re: [Openstack-operators] Managing quota for Nova local storage?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">We didn't come up with one. RAM on our HVs is the limiting factor since we don't run with memory overcommit, so the ability of people to run an HV out of disk space ended up being moot.
¯\_(</span><span style="font-family:"MS Mincho";color:black">ツ</span><span style="font-family:"Courier New";color:black">)_/¯
</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">Long term we would like to switch to being exclusively RBD-backed and get rid of local storage entirely, but that is Distant Future at best.</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"> </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">From: rovanleeuwen@ebay.com
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">Subject: Re: [Openstack-operators] Managing quota for Nova local storage?</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span class="bbscopedstyle7263624439947307"><span style="font-size:11.0pt;font-family:Calibri;color:black">Hi,</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:11.0pt;font-family:Calibri;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:11.0pt;font-family:Calibri;color:black">Found this thread in the archive so a bit of a late reaction.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:11.0pt;font-family:Calibri;color:black">We are hitting the same thing so I created a blueprint:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:11.0pt;font-family:Calibri;color:black"><a href="https://blueprints.launchpad.net/nova/+spec/nova-local-storage-quota">https://blueprints.launchpad.net/nova/+spec/nova-local-storage-quota</a>
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:11.0pt;font-family:Calibri;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:11.0pt;font-family:Calibri;color:black">If you guys already found a nice solution to this problem I’d like to hear it :)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:11.0pt;font-family:Calibri;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:11.0pt;font-family:Calibri;color:black">Robert van Leeuwen</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:11.0pt;font-family:Calibri;color:black">eBay - ECG</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:11.0pt;font-family:Calibri;color:black"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<b><span style="font-family:Calibri;color:black">From: </span></b><span style="font-family:Calibri;color:black">Warren Wang <warren@wangspeed.com><br>
<b>Date: </b>Wednesday, February 17, 2016 at 8:00 PM<br>
<b>To: </b>Ned Rhudy <erhudy@bloomberg.net><br>
<b>Cc: </b>"openstack-operators@lists.openstack.org" <openstack-operators@lists.openstack.org><br>
<b>Subject: </b>Re: [Openstack-operators] Managing quota for Nova local storage?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt;background:white">
<span style="color:black">We are in the same boat. Can't get rid of ephemeral for it's speed, and independence. I get it, but it makes management of all these tiny pools a scheduling and capacity nightmare.</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:black">Warren @ Walmart</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:black"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:black">On Wed, Feb 17, 2016 at 1:50 PM, Ned Rhudy (BLOOMBERG/ 731 LEX) <<a href="mailto:erhudy@bloomberg.net">erhudy@bloomberg.net</a>> wrote:</span><o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-family:"Courier New";color:black">The subject says it all - does anyone know of a method by which quota can be enforced on storage provisioned via Nova rather than Cinder? Googling around appears to indicate that this is not possible out of
the box (e.g., <a href="https://ask.openstack.org/en/question/8518/disk-quota-for-projects/">
https://ask.openstack.org/en/question/8518/disk-quota-for-projects/</a>). </span>
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-family:"Courier New";color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-family:"Courier New";color:black">The rationale is we offer two types of storage, RBD that goes via Cinder and LVM that goes directly via the libvirt driver in Nova. Users know they can escape the constraints of their volume quotas by using
the LVM-backed instances, which were designed to provide a fast-but-unreliable RAID 0-backed alternative to slower-but-reliable RBD volumes. Eventually users will hit their max quota in some other dimension (CPU or memory), but we'd like to be able to limit
based directly on how much local storage is used in a tenancy.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-family:"Courier New";color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-family:"Courier New";color:black">Does anyone have a solution they've already built to handle this scenario? We have a few ideas already for things we could do, but maybe somebody's already come up with something. (Social engineering on our
user base by occasionally destroying a random RAID 0 to remind people of their unsafety, while tempting, is probably not a viable candidate solution.)</span><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt;background:white">
<span style="color:black"><br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a></span><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:black"> </span><o:p></o:p></p>
</div>
</div>
</blockquote>
<p class="MsoNormal" style="background:white"><span style="font-family:Arial;color:black"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>