<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Matt,<br>
</p>
<div class="moz-cite-prefix">On 5/17/2018 3:18 PM, Matt Riedemann
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:f09b0d67-fd43-1617-e594-3326d8ceae30@gmail.com">On
5/17/2018 3:36 PM, Nadathur, Sundar wrote:
<br>
<blockquote type="cite">This applies only to the resources that
Nova handles, IIUC, which does not handle accelerators. The
generic method that Alex talks about is obviously preferable
but, if that is not available in Rocky, is the filter an option?
<br>
</blockquote>
<br>
If nova isn't creating accelerator resources managed by cyborg, I
have no idea why nova would be doing quota checks on those types
of resources. And no, I don't think adding a scheduler filter to
nova for checking accelerator quota is something we'd add either.
I'm not sure that would even make sense - the quota for the
resource is per tenant, not per host is it? The scheduler filters
work on a per-host basis.
<br>
</blockquote>
Can we not extend BaseFilter.filter_all() to get all the hosts in a
filter? <br>
<a class="moz-txt-link-freetext" href="https://github.com/openstack/nova/blob/master/nova/filters.py#L36">https://github.com/openstack/nova/blob/master/nova/filters.py#L36</a> <br>
<br>
I should have made it clearer that this putative filter will be
out-of-tree, and needed only till better solutions become available.
<blockquote type="cite"
cite="mid:f09b0d67-fd43-1617-e594-3326d8ceae30@gmail.com">
<br>
Like any other resource in openstack, the project that manages
that resource should be in charge of enforcing quota limits for
it.
<br>
</blockquote>
Agreed. Not sure how other projects handle it, but here's the
situation for Cyborg. A request may get scheduled on a compute node
with no intervention by Cyborg. So, the earliest check that can be
made today is in the selected compute node. A simple approach can
result in quota violations as in this example.<br>
<blockquote>Say there are 5 devices in a cluster. A tenant has a
quota of 4 and is currently using 3. That leaves 2 unused devices,
of which the tenant is permitted to use only one. But he may
submit two concurrent requests, and they may land on two different
compute nodes. The Cyborg agent in each node will see the current
tenant usage as 3 and let the request go through, resulting in
quota violation.<br>
</blockquote>
To prevent this, we need some kind of atomic update , like
SQLAlchemy's with_lockmode():<br>
<a class="moz-txt-link-freetext" href="https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#Pessimistic_Locking_-_SELECT_FOR_UPDATE">https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#Pessimistic_Locking_-_SELECT_FOR_UPDATE</a>
<br>
That seems to have issues, as documented in the link above. Also,
since every compute node does that, it would also serialize the
bringup of all instances with accelerators, across the cluster.<br>
<br>
If there is a better solution, I'll be happy to hear it.<br>
<br>
Thanks,<br>
Sundar<br>
<br>
<br>
<br>
<br>
</body>
</html>