<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">2016-11-03 4:52 GMT+08:00 Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 11/01/2016 10:14 AM, Alex Xu wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Currently we only update the resource usage with Placement API in the<br>
instance claim and the available resource update periodic task. But<br>
there is no claim for migration with placement API yet. This works is<br>
tracked by <a href="https://bugs.launchpad.net/nova/+bug/1621709" rel="noreferrer" target="_blank">https://bugs.launchpad.net/nov<wbr>a/+bug/1621709</a>. In newton, we<br>
only fix one bit which make the resource update periodic task works<br>
correctly, then it will auto-heal everything. For the migration claim<br>
part, that isn't the goal for newton release.<br>
<br>
So the first question is do we want to fix it in this release? If the<br>
answer is yes, there have a concern need to discuss.<br>
</blockquote>
<br></span>
Yes, I believe we should fix the underlying problem in Ocata. The underlying problem is what Sylvain brought up: live migrations do not currently use any sort of claim operation. The periodic resource audit is relied upon to essentially clean up the state of claimed resources over time, and as Chris points out in review comments on <a href="https://review.openstack.org/#/c/244489/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/244489/</a>, this leads to the scheduler operating on stale data and can lead to an increase in retry operations.<br>
<br>
This needs to be fixed before even attempting to address the issue you bring up with the placement API calls from the resource tracker.</blockquote><div><br></div><div>ok, let me see if I can help something at here.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In order to implement the drop of migration claim, the RT needs to<br>
remove allocation records on the specific RP(on the source/destination<br>
compute node). But there isn't any API can do that. The API about remove<br>
allocation records is 'DELETE /allocations/{consumer_uuid}', but it will<br>
delete all the allocation records for the consumer. So the initial<br>
fix(<a href="https://review.openstack.org/#/c/369172/" rel="noreferrer" target="_blank">https://review.openstack.o<wbr>rg/#/c/369172/</a>) adds new API 'DELETE<br>
/resource_providers/{rp_uuid}/<wbr>allocations/{consumer_id}'. But Chris Dent<br>
pointed out this against the original design. All the allocations for<br>
the specific consumer only can be dropped together.<br>
</blockquote>
<br></span>
Yes, and this is by design. Consumption of resources -- or the freeing thereof -- must be an atomic, transactional operation.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
There also have suggestion from Andrew, we can update all the allocation<br>
records for the consumer each time. That means the RT will build the<br>
original allocation records and new allocation records for the claim<br>
together, and put into one API. That API should be 'PUT<br>
/allocations/{consumer_uuid}'. Unfortunately that API doesn't replace<br>
all the allocation records for the consumer, it always amends the new<br>
allocation records for the consumer.<br>
</blockquote>
<br></span>
I see no reason why we can't change the behaviour of the `PUT /allocations/{consumer_uuid}` call to allow changing either the amounts of the allocated resources (a resize operation) or the set of resource provider UUIDs referenced in the allocations list (a move operation).<br>
<br>
For instance, let's say we have an allocation for an instance "i1" that is consuming 2 VCPU and 2048 MEMORY_MB on compute node "rpA", 50 DISK_GB on a shared storage pool "rpC".<br>
<br>
The allocations table would have the following records in it:<br>
<br>
resource_provider resource_class consumer used<br>
----------------- -------------- -------- ----<br>
rpA VCPU i1 2<br>
rpA MEMORY_MB i1 2048<br>
rpC DISK_GB i1 50<br>
<br>
Now, we need to migrate instance "i1" to compute node "rpB". The instance disk uses shared storage so the only allocation records we actually need to modify are the VCPU and MEMORY_MB records.<br></blockquote><div><br></div><div>yea, think about with shared storage, this makes sense a lot. Thanks for such detail explain at here!</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
We would create the following REST API call from the resource tracker on the destination node:<br>
<br>
PUT /allocations/i1<br>
{<br>
"allocations": [<br>
{<br>
"resource_provider": {<br>
"uuid": "rpB",<br>
},<br>
"resources": {<br>
"VCPU": 2,<br>
"MEMORY_MB": 2048<br>
}<br>
},<br>
{<br>
"resource_provider": {<br>
"uuid": "rpC",<br>
},<br>
"resources": {<br>
"DISK_GB": 50<br>
}<br>
}<br>
]<br>
}<br>
<br>
The placement service would receive that request payload and immediately grab any existing allocation records referencing consumer_uuid of "i1". It would notice that records referencing "rpA" (the source compute node) are no longer needed. It would notice that the DISK_GB allocation hasn't changed. And finally it would notice that there are new VCPU and MEMORY_MB records referring to a new resource provider "rpB" (the destination compute node).<br>
<br>
A single SQL transaction would be built that executes the following:<br>
<br>
BEGIN;<br>
<br>
# Grab the source and destination compute node provider generations<br>
# to protect against concurrent writes...<br>
$RPA_GEN := SELECT generation FROM resource_providers<br>
WHERE uuid = 'rpA';<br>
$RPB_GEN := SELECT generation FROM resource_providers<br>
WHERE uuid = 'rpB';<br>
<br>
# Delete the allocation records referring to the source for the VCPU<br>
# and MEMORY_MB resources<br>
DELETE FROM allocations<br>
WHERE consumer = 'i1'<br>
AND resource_provider = 'rpA'<br>
AND resource_class IN ('VCPU', 'MEMORY_MB');<br>
<br>
# Add allocation records referring to the destination for VCPU and<br>
# MEMORY_MB<br>
INSERT INTO allocations<br>
(resource_provider, resource_class, consumer, used)<br>
VALUES<br>
('rpB', 'VCPU', 'i1', 2),<br>
('rpb', 'MEMORY_MB', 'i1', 2048);<br>
<br>
# Update the resource provider generations and rollback the<br>
# transaction if any other writer modified the resource providers<br>
# in between the initial read time and here.<br>
UPDATE resource_providers<br>
SET generation = $RPA_GENERATION + 1<br>
WHERE uuid = 'rpA'<br>
AND generation = $RPA_GENERATION;<br>
<br>
IF ROWS_AFFECTED() == 0:<br>
ROLLBACK<br>
<br>
UPDATE resource_providers<br>
SET generation = $RPB_GENERATION + 1<br>
WHERE uuid = 'rpB'<br>
AND generation = $RPB_GENERATION;<br>
<br>
IF ROWS_AFFECTED() == 0:<br>
ROLLBACK<br>
<br>
COMMIT;<br>
<br>
In this way, we keep the API as is but simply handle move operations transparently to the caller. The caller simply expresses what they wish the allocation to look like with regards to which resource providers are having which resources consumed from, and the placement service ensures that these allocation records are written in an atomic fashion.<br>
<br>
Best,<br>
-jay<span class="im HOEnZb"><br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
So which directly we should go at here?<br>
</blockquote>
<br></span><div class="HOEnZb"><div class="h5">
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</div></div></blockquote></div><br></div></div>