<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:-webkit-standard;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Roboto Mono";
panose-1:0 0 0 9 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-AU" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Good morning Ha Noi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Okay, based on this it seems there maybe some sort of delay in Libvirt.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Have you got any logs from Libvirt?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Thanks,<br>
Karl.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Ha Noi <hanoi952022@gmail.com><br>
<b>Date: </b>Monday, 7 August 2023 at 6:57 pm<br>
<b>To: </b>Karl Kloppenborg <kkloppenborg@rwts.com.au><br>
<b>Cc: </b>openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org><br>
<b>Subject: </b>Re: [openstack][nova] Slow provision 1 instance .<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Karl,<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">time rbd-nbd --id cinder map SSD1/volume-f2ff322b-c9f4-4efe-9b6e-d80a9d0aa043_ERR<br>
/dev/nbd0<br>
<br>
real 0m0.111s<br>
user 0m0.028s<br>
sys 0m0.005s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">The time mapping is quite fast.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks,<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">On Mon, Aug 7, 2023 at 11:27 AM Karl Kloppenborg <<a href="mailto:kkloppenborg@rwts.com.au">kkloppenborg@rwts.com.au</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">Hi Ha Noi,</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">Correct, try and take the same volume and directly map it on the server, see if it hangs for any period of
time.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">--Karl.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Ha Noi <<a href="mailto:hanoi952022@gmail.com" target="_blank">hanoi952022@gmail.com</a>><br>
<b>Date: </b>Monday, 7 August 2023 at 1:11 pm<br>
<b>To: </b>Karl Kloppenborg <<a href="mailto:kkloppenborg@rwts.com.au" target="_blank">kkloppenborg@rwts.com.au</a>><br>
<b>Cc: </b><a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a> <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject: </b>Re: [openstack][nova] Slow provision 1 instance .</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Hi Karl,<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Do you mean try rbd-nbd map a volume from ceph to compute node?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Thanks,<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">On Mon, Aug 7, 2023 at 8:58 AM Karl Kloppenborg <<a href="mailto:kkloppenborg@rwts.com.au" target="_blank">kkloppenborg@rwts.com.au</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Hi Ha Noi,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">From those logs it appears that the lag is in the libvirt section of the block device connecting and latching to the compute node.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">If you try and rbd mount manually a drive from your ceph cluster, do you see a long delay in this as well?<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">Kind Regards,</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">--</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">Karl Kloppenborg,
</span></b><i><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">(BCompSc, CNCF-[KCNA, CKA, CKAD], LFCE, CompTIA Linux+ XK0-004)</span></i><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:-webkit-standard;color:black">Managing Director, Invention Labs.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Ha Noi <<a href="mailto:hanoi952022@gmail.com" target="_blank">hanoi952022@gmail.com</a>><br>
<b>Date: </b>Monday, 7 August 2023 at 11:38 am<br>
<b>To: </b><a href="mailto:kkloppenborg@rwts.com.au" target="_blank">kkloppenborg@rwts.com.au</a> <<a href="mailto:kkloppenborg@rwts.com.au" target="_blank">kkloppenborg@rwts.com.au</a>><br>
<b>Cc: </b><a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a> <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject: </b>Re: [openstack][nova] Slow provision 1 instance .</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Hi Karl,<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">The volume creation time is 10 seconds. Below is the log in nova-compute. I would like to optimize nova-compute first .<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">3414:2023-08-07 08:33:08.436 8 INFO nova.compute.claims [req-33bf9cfd-38c5-40c3-aceb-2c4c0bfa986c 125406d260d24c77aca1b09206825437 f64e241d000441d292b91e85138e325c
- default default] [instance: c54db6fe-7bf0-4ca3-8167-20ec2763b79f] Claim successful on node compute-004<br>
3415:2023-08-07 08:33:08.702 8 INFO nova.virt.libvirt.driver [req-33bf9cfd-38c5-40c3-aceb-2c4c0bfa986c 125406d260d24c77aca1b09206825437 f64e241d000441d292b91e85138e325c - default default] [instance: c54db6fe-7bf0-4ca3-8167-20ec2763b79f] Ignoring supplied device
name: /dev/vda. Libvirt can't honour user-supplied dev names<br>
3416:2023-08-07 08:33:08.815 8 INFO nova.virt.block_device [req-33bf9cfd-38c5-40c3-aceb-2c4c0bfa986c 125406d260d24c77aca1b09206825437 f64e241d000441d292b91e85138e325c - default default] [instance: c54db6fe-7bf0-4ca3-8167-20ec2763b79f] Booting with volume 37d1487e-9118-4231-9908-fb662b626977
at /dev/vda<br>
3417:2023-08-07 08:33:34.269 8 INFO nova.virt.libvirt.driver [req-33bf9cfd-38c5-40c3-aceb-2c4c0bfa986c 125406d260d24c77aca1b09206825437 f64e241d000441d292b91e85138e325c - default default] [instance: c54db6fe-7bf0-4ca3-8167-20ec2763b79f] Creating image<br>
3418:2023-08-07 08:33:35.967 8 INFO os_vif [req-33bf9cfd-38c5-40c3-aceb-2c4c0bfa986c 125406d260d24c77aca1b09206825437 f64e241d000441d292b91e85138e325c - default default] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:1e:06:ae,bridge_name='qbr2ea5e4e0-a9',has_traffic_filtering=True,id=2ea5e4e0-a934-404d-ac4c-a4deadf2aa73,network=Network(a4e1151a-9671-4fb6-b620-66b80d7dce8d),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tap2ea5e4e0-a9')<br>
3419:2023-08-07 08:33:37.763 8 INFO os_vif [req-33bf9cfd-38c5-40c3-aceb-2c4c0bfa986c 125406d260d24c77aca1b09206825437 f64e241d000441d292b91e85138e325c - default default] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:b0:e6:12,bridge_name='qbre636e020-ec',has_traffic_filtering=True,id=e636e020-ec99-47d4-bad3-9c87c65e080f,network=Network(5f7989ab-cb9d-493b-a8c5-edf688782f81),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tape636e020-ec')<br>
3425:2023-08-07 08:33:45.725 8 INFO nova.compute.manager [req-33bf9cfd-38c5-40c3-aceb-2c4c0bfa986c 125406d260d24c77aca1b09206825437 f64e241d000441d292b91e85138e325c - default default] [instance: c54db6fe-7bf0-4ca3-8167-20ec2763b79f] Took 11.46 seconds to spawn
the instance on the hypervisor.<br>
3459:2023-08-07 08:34:02.909 8 INFO nova.compute.manager [req-33bf9cfd-38c5-40c3-aceb-2c4c0bfa986c 125406d260d24c77aca1b09206825437 f64e241d000441d292b91e85138e325c - default default] [instance: c54db6fe-7bf0-4ca3-8167-20ec2763b79f] Took 61.40 seconds to build
instance.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">As you can see, the first log, instance was claimed on node compute 004 at 08:33:08, Then it takes 1 minute (04:34:02) to build instance.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Thanks <o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">On Fri, Aug 4, 2023 at 10:25 PM Ha Noi <<a href="mailto:hanoi952022@gmail.com" target="_blank">hanoi952022@gmail.com</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Hi everyone,<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">We have a openstack with one region and more than 100 compute nodes. We are using ceph for block storage. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">I don't know why my instance was provisioned too slow: more than 1 minutes. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Nova compute log: <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt;vertical-align:baseline">
<span style="font-size:9.0pt;font-family:"Roboto Mono";color:#343741;letter-spacing:-.05pt;border:none windowtext 1.0pt;padding:0cm">Took 56.47 seconds to build instance.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Roboto Mono";color:#343741;letter-spacing:-.05pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#343741;letter-spacing:-.05pt">So I would like to optimize and speed up instance build time. </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#343741;letter-spacing:-.05pt">Thanks and Best Regards</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>