<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:inherit;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.vote-count-post
{mso-style-name:vote-count-post;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle27
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Moshe,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thank you. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Dear All,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Has anyone worked with intel NICs to enable SRIOV in Openstack. If yes, please if you could help me in configuring the NIC to be made suitable for Openstack to pick up. Currently, I am facing issues
with Libvirt failing to list the node. With driver i40evf running, it lists the interfaces. However, if I disable the i40evf driver it does not list the interfaces in nodedev-list. Please if someone could verify this as right behaviour or not.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Kind Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Navdeep<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Moshe Levi [mailto:moshele@mellanox.com]
<br>
<b>Sent:</b> 29 January 2018 12:00<br>
<b>To:</b> Navdeep Uniyal <navdeep.uniyal@bristol.ac.uk>; openstack@lists.openstack.org<br>
<b>Subject:</b> RE: [Openstack] Openstack SRIOV and PCI passthrough error while creating instance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Hi Nadepp,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">I am not familiar with Intel NIC (I work at Mellanox)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Maybe the intel folks in the mailing list can help you.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Navdeep Uniyal [<a href="mailto:navdeep.uniyal@bristol.ac.uk">mailto:navdeep.uniyal@bristol.ac.uk</a>]
<br>
<b>Sent:</b> Monday, January 29, 2018 1:58 PM<br>
<b>To:</b> Moshe Levi <<a href="mailto:moshele@mellanox.com">moshele@mellanox.com</a>>;
<a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
<b>Subject:</b> RE: [Openstack] Openstack SRIOV and PCI passthrough error while creating instance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal">Hi Moshe,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you for your response.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am creating VFs on intel X710 NIC. As per the documentation I need to disable ‘i40evf driver’ on the host machine to get the VMs running with the VFs however, on doing so I am not getting the VF list while doing ‘virsh nodedev-list’.
Could you please advise if this driver is required to be running or not.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best Regards,<o:p></o:p></p>
<p class="MsoNormal">Navdeep<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Moshe Levi [<a href="mailto:moshele@mellanox.com">mailto:moshele@mellanox.com</a>]
<br>
<b>Sent:</b> 29 January 2018 09:30<br>
<b>To:</b> Navdeep Uniyal <<a href="mailto:navdeep.uniyal@bristol.ac.uk">navdeep.uniyal@bristol.ac.uk</a>>;
<a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
<b>Subject:</b> RE: [Openstack] Openstack SRIOV and PCI passthrough error while creating instance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Can you try to restart libvirtd and query again "virsh nodedev-list"<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Navdeep Uniyal [<a href="mailto:navdeep.uniyal@bristol.ac.uk">mailto:navdeep.uniyal@bristol.ac.uk</a>]
<br>
<b>Sent:</b> Monday, January 29, 2018 10:32 AM<br>
<b>To:</b> Moshe Levi <<a href="mailto:moshele@mellanox.com">moshele@mellanox.com</a>>;
<a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
<b>Subject:</b> RE: [Openstack] Openstack SRIOV and PCI passthrough error while creating instance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal">Hi Moshe,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you very much for your response. I could see the list of nodes using virsh:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">net_enp129s0f0_3c_fd_fe_a9_37_c0<o:p></o:p></p>
<p class="MsoNormal">net_enp129s0f1_3c_fd_fe_a9_37_c1<o:p></o:p></p>
<p class="MsoNormal">net_enp129s2_36_a5_18_f7_41_c6<o:p></o:p></p>
<p class="MsoNormal">net_enp129s2f1_96_b0_b3_37_cf_c5<o:p></o:p></p>
<p class="MsoNormal">net_enp129s2f2_0e_0c_63_76_07_0f<o:p></o:p></p>
<p class="MsoNormal">net_enp129s2f3_36_ca_41_10_7f_62<o:p></o:p></p>
<p class="MsoNormal">net_enp129s2f4_f2_79_ee_75_38_a9<o:p></o:p></p>
<p class="MsoNormal">net_enp129s2f5_2a_a1_c6_02_55_11<o:p></o:p></p>
<p class="MsoNormal">net_enp129s2f6_16_e3_c4_01_8c_6d<o:p></o:p></p>
<p class="MsoNormal">net_enp129s2f7_66_56_63_d8_a8_ad<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">these does not have the node <span style="font-size:11.5pt;font-family:inherit;color:#242729">
'net_enp129s2_b2_87_6e_13_a1_5e' listed. Could you please help me finding out what could be the reason behind this mismatch.</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Kind Regards,<o:p></o:p></p>
<p class="MsoNormal">Navdeep Uniyal<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Moshe Levi [<a href="mailto:moshele@mellanox.com">mailto:moshele@mellanox.com</a>]
<br>
<b>Sent:</b> 29 January 2018 06:50<br>
<b>To:</b> Navdeep Uniyal <<a href="mailto:navdeep.uniyal@bristol.ac.uk">navdeep.uniyal@bristol.ac.uk</a>>;
<a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
<b>Subject:</b> RE: [Openstack] Openstack SRIOV and PCI passthrough error while creating instance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Hi Navdeep,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">The errors you are pointing out shouldn't cause error while creating instance with SR-IOV/PCI passthrough.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">They are part of the SR-IOV nic capabilities feature which is not completed yet.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Libvirt "virsh nodedev-list" command will show you the list of nic and this method [1] will look it in the hypervisor.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">For some reason you have a mismatch here. You need to compare them. Anyway if there is a mismatch it just won't report the capabilities of the nic.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">As I mentioned above I think this is not the issue for the error while creating instance
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">[1] <a href="https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fnova%2Fblob%2F91f7a999988b3f857d738d39984117e6c514cbec%2Fnova%2Fpci%2Futils.py%23L196-L217&data=02%7C01%7Cmoshele%40mellanox.com%7C64da232237d54639b39208d566f2cc83%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636528115363097991&sdata=9warkJI5zs0l57YKgpBrXTUA5IO7lN1xWAAKxDo%2FSy0%3D&reserved=0">
https://github.com/openstack/nova/blob/91f7a999988b3f857d738d39984117e6c514cbec/nova/pci/utils.py#L196-L217</a>
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Navdeep Uniyal [<a href="mailto:navdeep.uniyal@bristol.ac.uk">mailto:navdeep.uniyal@bristol.ac.uk</a>]
<br>
<b>Sent:</b> Sunday, January 28, 2018 10:58 PM<br>
<b>To:</b> <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
<b>Subject:</b> [Openstack] Openstack SRIOV and PCI passthrough error while creating instance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal">Dear All,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am getting an error while creating SRIOV enabled instances following the guide from Openstack pike
<a href="https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.openstack.org%2Fneutron%2Fpike%2Fadmin%2Fconfig-sriov.html&data=02%7C01%7Cmoshele%40mellanox.com%7C7035d805a74d45ff0bbe08d56692fd3e%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636527703870300090&sdata=Oljh3%2Ff1B%2FLCsNxHIUkS%2FgLzTBkpOY3XaWBixYKWOaQ%3D&reserved=0">
https://docs.openstack.org/neutron/pike/admin/config-sriov.html</a> <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" style="background:white;border-collapse:collapse">
<tbody>
<tr>
<td valign="top" style="padding:0cm 11.25pt 0cm 0cm"></td>
<td valign="top" style="padding:0cm 0cm 0cm 0cm;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch: inherit;font-size:inherit;line-height:inherit">
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm;vertical-align:baseline;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch: inherit;font-size:inherit;line-height:inherit">
<span style="font-size:11.5pt;font-family:inherit;color:#242729">I want to start an instance on openstack pike with SRIOV enabled NICs. However, I am getting a Libvirt error regarding the node name. The error looks weird as the node name is not matching the
interface name on the host machine or in the configuration files.<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm;vertical-align:baseline;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch: inherit;font-size:inherit;line-height:inherit">
<span style="font-size:11.5pt;font-family:inherit;color:#242729">2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager [req-caa92f1d-5ac1-402d-a8bc-b08ab350a21f - - - - -] Error updating resources for node jupiter.: libvirtError: Node device not found: no
node device with matching name 'net_enp129s2_b2_87_6e_13_a1_5e' 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager Traceback (most recent call last): 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
line 6696, in update_available_resource_for_node 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager rt.update_available_resource(context, nodename) 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py",
line 641, in update_available_resource 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager resources = self.driver.get_available_resource(nodename) 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py",
line 5857, in get_available_resource 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5621, in _get_pci_passthrough_devices
2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager for name in dev_names: 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5582, in _get_pcidev_info 2018-01-28 20:40:11.416
2953 ERROR nova.compute.manager device['label'] = 'label_%(vendor_id)s_%(product_id)s' % device 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5553, in _get_device_capabilities
2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager pcinet_info = self._get_pcinet_info(address) 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5496, in _get_pcinet_info
2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager virtdev = self._host.device_lookup_by_name(devname) 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 845, in device_lookup_by_name
2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager return self.get_connection().nodeDeviceLookupByName(name) 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit 2018-01-28
20:40:11.416 2953 ERROR nova.compute.manager result = proxy_call(self._autowrap, f, *args, **kwargs) 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call 2018-01-28 20:40:11.416
2953 ERROR nova.compute.manager rv = execute(f, *args, **kwargs) 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager
six.reraise(c, e, tb) 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager rv = meth(*args, **kwargs) 2018-01-28 20:40:11.416
2953 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/libvirt.py", line 4177, in nodeDeviceLookupByName 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager if ret is None:raise libvirtError('virNodeDeviceLookupByName() failed', conn=self)
2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager libvirtError: Node device not found: no node device with matching name 'net_enp129s2_b2_87_6e_13_a1_5e' 2018-01-28 20:40:11.416 2953 ERROR nova.compute.manager<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm;vertical-align:baseline;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch: inherit;font-size:inherit;line-height:inherit">
<span style="font-size:11.5pt;font-family:inherit;color:#242729">The correct interface name is enp129s0f0. However, I am getting the node name as net_enp129s2_b2_87_6e_13_a1_5e' which i believe is the reason behind the failure of vm creation on openstack. Please
if someone could help me understand how the node name is passed on to the Libvirt from openstack or how can I resolve this issue.<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal">Navdeep<o:p></o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>