<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Apr 10, 2016 at 10:07 AM, Moshe Levi <span dir="ltr"><<a href="mailto:moshele@mellanox.com" target="_blank">moshele@mellanox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Miguel Angel Ajo Pelayo [mailto:<a href="mailto:majopela@redhat.com" target="_blank">majopela@redhat.com</a>]
<br>
<b>Sent:</b> Friday, April 08, 2016 4:17 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<b>Subject:</b> [openstack-dev] [neutron] [nova] scheduling bandwidth resources / NIC_BW_KB resource class<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
<div><span class="">
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-left:.5in">Hi!, <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> In the context of [1] (generic resource pools / scheduling in nova) and [2] (minimum bandwidth guarantees -egress- in neutron), I had a talk a few weeks ago with Jay Pipes, <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
</span><div><span class="">
<p class="MsoNormal" style="margin-left:.5in"> The idea was leveraging the generic resource pools and scheduling mechanisms defined in [1] to find the right hosts and track the total available bandwidth per host (and per host "physical network"), something
in neutron (still to be defined where) would notify the new API about the total amount of "NIC_BW_KB" available on every host/physnet.</p></span></div></div></div></div></blockquote><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><div><div><span class=""><p class="MsoNormal" style="margin-left:.5in"><u></u><u></u></p>
</span><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I believe that NIC bandwidth can be taken from Libvirt see [4] and the only piece that is missing is to tell nova the mapping of physnet to network interface
name. (In case of SR-IOV this is already known) <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I see bandwidth (speed) as one of many capabilities of NIC, therefore I think we should take all of them in the same way in this case libvirt. I was think of
adding a new NIC as new resource to nova.</span></p></div></div></div></div></blockquote><div><br></div><div>Yes, at the low level, thats one way to do it. We may need neutron agents or plugins to collect such information, since, in some cases one devices will be tied to one physical network, other devices will be tied to other physical networks, or even some devices could be connected to the same physnet. In some cases, connectivity depends on L3 tunnels, and in that case, bandwidth calculation is more complicated (depending on routes, etc.. -I'm not even looking at that case yet-)</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><div><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">[4] - <device><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <name>net_enp129s0_e4_1d_2d_2d_8c_41</name><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <path>/sys/devices/pci0000:80/0000:80:01.0/0000:81:00.0/net/enp129s0</path><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <parent>pci_0000_81_00_0</parent><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <capability type='net'><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <interface>enp129s0</interface><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <address>e4:1d:2d:2d:8c:41</address><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <link speed='40000' state='up'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <feature name='rx'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <feature name='tx'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <feature name='sg'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <feature name='tso'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <feature name='gso'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <feature name='gro'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <feature name='rxvlan'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <feature name='txvlan'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <feature name='rxhash'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <feature name='rdma'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> <capability type='80203'/><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </capability><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"></device><u></u><u></u></span></p>
</div><span class="">
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> That part is quite clear to me,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> From [1] I'm not sure which blueprint introduces the ability to schedule based on the resource allocation/availability itself, ("resource-providers-scheduler" seems more like an optimization to the schedule/DB
interaction, right?)<span style="color:#1f497d"><u></u><u></u></span></p>
</div>
</span><div>
<p class="MsoNormal"><span style="color:#1f497d">My understating is that the resource provider blueprint is just a rough filter of compute nodes before passing them to the scheduler filters. The existing filters here [6] will do the accurate filtering of resources.</span>
<span style="color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">see [5] <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">[5] - <a href="http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-04-04.log.html#t2016-04-04T16:24:10" target="_blank">http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-04-04.log.html#t2016-04-04T16:24:10</a> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">[6] -<b> </b><a href="http://docs.openstack.org/developer/nova/filter_scheduler.html" target="_blank"><span style="color:#1f497d;text-decoration:none">http://docs.openstack.org/developer/nova/filter_scheduler.html</span></a><b>
<u></u><u></u></b></span></p>
<p class="MsoNormal"><u></u> </p></div></div></div></div></blockquote><div><br></div><div>Thanks, yes, if those filters can operate on the generic resource pools, then, great, we will just need to write the right filters.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><div><div><p class="MsoNormal"><u></u></p>
</div><span class="">
<div>
<p class="MsoNormal" style="margin-left:.5in"> And, that brings me to another point: at the moment of filtering hosts, nova I guess, will have the neutron port information, it has to somehow identify if the port is tied to a minimum bandwidth QoS policy.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
</span><div><span class="">
<p class="MsoNormal" style="margin-left:.5in"> That would require identifying that the port has a "qos_policy_id" attached to it, and then, asking neutron for the specific QoS policy [3], then look out for a minimum bandwidth rule (still to be defined),
and extract the required bandwidth from it.<u></u><u></u></p>
</span><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I am not sure if that is the correct way to do it, but you can create NIC bandwidth filter (or NIC capabilities filter) and in it you can implement the way
to retrieve Qos policy information by using neutron client.</span></p></div></div></div></div></blockquote><div><br></div><div>That's my concern, that logic would have to live on the nova side, again, and it's tightly couple to the neutron models. I'd be glad to find a way to uncouple nova from that as much as possible. And, even better if we could find a way to avoid the need for nova to retrieve policies as it discovers ports.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><div><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u><u></u></span></p>
</div><span class="">
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> That moves, again some of the responsibility to examine and understand external resources to nova.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> Could it make sense to make that part pluggable via stevedore?, so we would provide something that takes the "resource id" (for a port in this case) and returns the requirements translated to resource classes
(NIC_BW_KB in this case).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Best regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Miguel Ángel Ajo<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">[1] <a href="http://lists.openstack.org/pipermail/openstack-dev/2016-February/086371.html" target="_blank">
http://lists.openstack.org/pipermail/openstack-dev/2016-February/086371.html</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">[2] <a href="https://bugs.launchpad.net/neutron/+bug/1560963" target="_blank">
https://bugs.launchpad.net/neutron/+bug/1560963</a> <u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">[3] <a href="http://developer.openstack.org/api-ref-networking-v2-ext.html#showPolicy" target="_blank">http://developer.openstack.org/api-ref-networking-v2-ext.html#showPolicy</a><u></u><u></u></p>
</div>
</span></div>
</div>
</div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>