<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
h1
{mso-style-priority:9;
mso-style-link:"Heading 1 Char";
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:24.0pt;
font-family:SimSun;
font-weight:bold;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.Heading1Char
{mso-style-name:"Heading 1 Char";
mso-style-priority:9;
mso-style-link:"Heading 1";
font-family:SimSun;
font-weight:bold;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Ian, thanks for your reply. Please check my response prefix with ‘yjiang5’.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">--jyh<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Ian Wells [mailto:ijw.ubuntu@cack.org.uk]
<br>
<b>Sent:</b> Friday, January 10, 2014 4:08 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [nova] [neutron] PCI pass-through network support<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On 10 January 2014 07:40, Jiang, Yunhong <<a href="mailto:yunhong.jiang@intel.com" target="_blank">yunhong.jiang@intel.com</a>> wrote:<o:p></o:p></span></p>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Robert, sorry that I’m not fan of * your group * term. To me, *your group” mixed two
thing. It’s an extra property provided by configuration, and also it’s a very-not-flexible mechanism to select devices (you can only select devices based on the ‘group name’ property).</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">It is exactly that. It's 0 new config items, 0 new APIs, just an extra tag on the whitelists that are already there (although the proposal suggests changing the name of them to be more descriptive of what they now do).
And you talk about flexibility as if this changes frequently, but in fact the grouping / aliasing of devices almost never changes after installation, which is, not coincidentally, when the config on the compute nodes gets set up.<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">1)</span><span lang="EN-US" style="font-size:7.0pt;color:#1F497D">
</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">A dynamic group is much better. For example, user may want to select GPU device based on vendor id, or based on vendor_id+device_id. In another word, user want
to create group based on vendor_id, or vendor_id+device_id and select devices from these group. John’s proposal is very good, to provide an API to create the PCI flavor(or alias). I prefer flavor because it’s more openstack style.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US">I disagree with this. I agree that what you're saying offers a more flexibilibility after initial installation but I have various issues with it.<o:p></o:p></span></p>
<h1 style="margin:0cm;margin-bottom:.0001pt;line-height:19.8pt;background:white;vertical-align:baseline">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D;font-weight:normal">[yjiang5] I think you talking is mostly about white list, instead of PCI flavor. PCI flavor is more about PCI request, like I want to have a device
with “vendor_id = cisco, device_id= 15454E”, or ‘vendor_id=intel device_class=nic’ , ( because the image have the driver for all Intel NIC card
</span><span lang="EN-US" style="font-size:10.5pt;font-family:Wingdings;color:#1F497D;font-weight:normal">J</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D;font-weight:normal"> ). While whitelist is to decide
the device that is assignable in a host.<o:p></o:p></span></h1>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">“<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
This is directly related to the hardware configuation on each compute node. For (some) other things of this nature, like provider networks, the compute node is the only thing that knows what it has attached to it, and it is the store (in configuration) of
that information. If I add a new compute node then it's my responsibility to configure it correctly on attachment, but when I add a compute node (when I'm setting the cluster up, or sometime later on) then it's at that precise point that I know how I've attached
it and what hardware it's got on it. Also, it's at this that point in time that I write out the configuration file (not by hand, note; there's almost certainly automation when configuring hundreds of nodes so arguments that 'if I'm writing hundreds of config
files one will be wrong' are moot). <br>
<br>
I'm also not sure there's much reason to change the available devices dynamically after that, since that's normally an activity that results from changing the physical setup of the machine which implies that actually you're going to have access to and be able
to change the config as you do it. John did come up with one case where you might be trying to remove old GPUs from circulation, but it's a very uncommon case that doesn't seem worth coding for, and it's still achievable by changing the config and restarting
the compute processes.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">[yjiag5] I totally agree with you that whitelist is static defined when provision. I just want to separate the information
of ‘provider network’ to another configuration (like extra information). Whitelist is just white list to decide the device assignable. The provider network is information of the device, it’s not in the scope of the white list.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">This also reduces the autonomy of the compute node in favour of centralised tracking, which goes against the 'distributed where possible' philosophy of Openstack.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">Finally, you're not actually removing configuration from the compute node. You still have to configure a whitelist there; in the grouping design you also have to configure grouping (flavouring)
on the control node as well. The groups proposal adds one extra piece of information to the whitelists that are already there to mark groups, not a whole new set of config lines.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">[yjiang5] Still, while list is to decide the device assignable, not to provide device information. We should mixed
functionality to the configuration. If it’s ok, I simply want to discard the ‘group’ term
</span><span lang="EN-US" style="font-size:10.5pt;font-family:Wingdings;color:#1F497D">J</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"> The nova PCI flow is simple, compute node provide PCI device (based
on white list), the scheduler track the PCI device information (abstracted as pci_stats for performance issue), the API provide method that user specify the device they wanted (the PCI flavor). Current implementation need enhancement on each step of the flow,
but I really see no reason to have the “Group”. Yes, the ‘PCI flavor’ in fact create group based on PCI property, but it’s better to be expressed as flavor.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">To compare scheduling behaviour:<br>
<br>
If I need 4G of RAM, each compute node has reported its summary of free RAM to the scheduler. I look for a compute node with 4G free, and filter the list of compute nodes down. This is a query on n records, n being the number of compute nodes. I schedule
to the compute node, which then confirms it does still have 4G free and runs the VM or rejects the request.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">If I need 3 PCI devices and use the current system, each machine has reported its device allocations to the scheduler. With SRIOV multiplying up the number of available devices, it's reporting
back hundreds of records per compute node to the schedulers, and the filtering activity is a 3 queries on n * number of PCI devices in cloud records, which could easily end up in the tens or even hundreds of thousands of records for a moderately sized cloud.
There compute node also has a record of its device allocations which is also checked and updated before the final request is run.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">If I need 3 PCI devices and use the groups system, each machine has reported its device *summary* to the scheduler. With SRIOV multiplying up the number of available devices, it's still reporting
one or a small number of categories, i.e. { net: 100}. The difficulty of scheduling is a query on num groups * n records - fewer, in fact, if some machines have no passthrough devices.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">[yjiang5] That’s the reason we have the pci_stats. The PCI stats is a * summary * for PCI devices information based
on *<b>selected</b>* PCI property, like vendor_id, device_id. If we assume the all VFs has the same vendor_id/device_id, it will becomes in fact only one entry in the pci_stats! However, we still keep the detailed information like vendor_id/device_id in the
scheduler for decision making, instead of the opaque ‘group name’. And with a configuration to select which property to be used for the pci_stats, like ‘vendor_id’ only, or ‘vendor_id/device_id’, it’s much flexible. And if extend the Nova PCI to have user
defined property, you can simply add property like ‘net’ to all your assignable devices, and then configure the ‘net’ as the only property to get the pci_stats, that’s exactly the implementation as your idea !<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">You can see that there's quite a cost to be paid for having those flexible alias APIs.<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">4)</span><span lang="EN-US" style="font-size:7.0pt;color:#1F497D">
</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">IMHO, the core for nova PCI support is *<b>PCI property</b>*. The property means not only generic PCI devices like vendor id, device id, device type, compute
specific property like BDF address, the adjacent switch IP address, but also user defined property like nuertron’s physical net name etc. And then, it’s about how to get these property, how to select/group devices based on the property, how to store/fetch
these properties.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
The thing about this is that you don't always, or even often, want to select by property. Some of these properties are just things that you need to tell Neutron, they're not usually keys for scheduling.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">[yjiang5] Yes, that’s the reason of pci_stats, which use only selected property for scheduling. But I don’t want to fixed the selected property
to be only ‘groupname’!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">--jyh<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>