<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Eddie,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">
<o:p></o:p></span></p>
<pre><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Looking on the your nova database after the delete looks correct to me. <o:p></o:p></span></pre>
<pre><span style="color:black">| created_at | updated_at | deleted_at | deleted | id<o:p></o:p></span></pre>
<pre><span style="color:black">| 2017-06-21 00:56:06 | 2017-07-07 02:27:16 | NULL | 0 | 2<o:p></o:p></span></pre>
<pre><span style="color:black">| 2017-07-07 01:42:48 | 2017-07-07 02:13:14 | 2017-07-07 02:13:42 | 9 | 9 <o:p></o:p></span></pre>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">See that the second row has deleted_at timestamp and deleted with no zero value (the id of the row). Nova is doing soft delete which is just marking the row
as deleted but not actually deleting it from nova pci_devices table.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">See [1] and [2]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">There is a bug with the pci_devices in a scenario when we can delete allocated pci device e.g. if pci.passthrough_whitelist is</span><span style="font-family:"Courier New";color:black;background:white">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">changed commit [3] try to resolve.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[1] -
<a href="https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/models.py#L142-L150">
https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/models.py#L142-L150</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[2] -
<a href="https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/models.py#L1411">
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/models.py#L1411</a>
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[3-] -
<a href="https://review.openstack.org/#/c/426243/">https://review.openstack.org/#/c/426243/</a>
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Eddie Yen [mailto:missile0407@gmail.com]
<br>
<b>Sent:</b> Tuesday, July 11, 2017 3:18 AM<br>
<b>To:</b> Jay Pipes <jaypipes@gmail.com><br>
<b>Cc:</b> openstack@lists.openstack.org<br>
<b>Subject:</b> Re: [Openstack] [nova] Database not delete PCI info after device is removed from host and nova.conf<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Roger that,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I may going to report this bug on the OpenStack Compute (Nova) Launchpad to see what happen.<br>
<br>
Anyway, thanks for ur help, really appreciate.<br>
<br>
<br>
Eddie.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">2017-07-11 8:12 GMT+08:00 Jay Pipes <<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Unfortunately, Eddie, I'm not entirely sure what is going on with your situation. According to the code, the non-existing PCI device should be removed from the pci_devices table when the PCI manager notices the PCI device is no longer on
the local host...<br>
<br>
On 07/09/2017 08:36 PM, Eddie Yen wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Hi there,<br>
<br>
Does the information already enough or need additional items?<br>
<br>
Thanks,<br>
Eddie.<br>
<br>
2017-07-07 10:49 GMT+08:00 Eddie Yen <<a href="mailto:missile0407@gmail.com" target="_blank">missile0407@gmail.com</a> <mailto:<a href="mailto:missile0407@gmail.com" target="_blank">missile0407@gmail.com</a>>>:<br>
<br>
Sorry,<br>
<br>
Re-new the nova-compute log after remove "1002:68c8" and restart<br>
nova-compute.<br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2FqUCOX09jyeMydoYHc8Oz%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098573075&sdata=brxkAv3AgO%2BwpwPXow5SY%2By0rGZ%2B7STTbEfm3gH1KSM%3D&reserved=0" target="_blank">
http://paste.openstack.org/show/qUCOX09jyeMydoYHc8Oz/</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2FqUCOX09jyeMydoYHc8Oz%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098573075&sdata=brxkAv3AgO%2BwpwPXow5SY%2By0rGZ%2B7STTbEfm3gH1KSM%3D&reserved=0" target="_blank">http://paste.openstack.org/show/qUCOX09jyeMydoYHc8Oz/</a>><br>
<br>
2017-07-07 10:37 GMT+08:00 Eddie Yen <<a href="mailto:missile0407@gmail.com" target="_blank">missile0407@gmail.com</a><br>
<mailto:<a href="mailto:missile0407@gmail.com" target="_blank">missile0407@gmail.com</a>>>:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
<br>
Hi Jay,<br>
<br>
Below are few logs and information you may want to check.<br>
<br>
<br>
<br>
I wrote GPU inforamtion into nova.conf like this.<br>
<br>
pci_passthrough_whitelist = [{ "product_id":"0ff3",<br>
"vendor_id":"10de"}, { "product_id":"68c8", "vendor_id":"1002"}]<br>
<br>
pci_alias = [{ "product_id":"0ff3", "vendor_id":"10de",<br>
"device_type":"type-PCI", "name":"k420"}, { "product_id":"68c8",<br>
"vendor_id":"1002", "device_type":"type-PCI", "name":"v4800"}]<br>
<br>
<br>
Then restart the services.<br>
<br>
nova-compute log when insert new GPU device info into nova.conf<br>
and restart service:<br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2Fz015rYGXaxYhVoafKdbx%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=Jc1%2B7Uexui%2FFfEL%2FdADTp6tVa9ssIBPGabGwA85Qm2E%3D&reserved=0" target="_blank">
http://paste.openstack.org/show/z015rYGXaxYhVoafKdbx/</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2Fz015rYGXaxYhVoafKdbx%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=Jc1%2B7Uexui%2FFfEL%2FdADTp6tVa9ssIBPGabGwA85Qm2E%3D&reserved=0" target="_blank">http://paste.openstack.org/show/z015rYGXaxYhVoafKdbx/</a>><br>
<br>
Strange is, the log shows that resource tracker only collect<br>
information of new setup GPU, not included the old one.<br>
<br>
<br>
But If I do some actions on the instance contained old GPU, the<br>
tracker will get both GPU.<br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2F614658%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=EvEVi1mhEAbVLK7NQppVJX8i7aqkgCtwbH8GRFr81Fo%3D&reserved=0" target="_blank">
http://paste.openstack.org/show/614658/</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2F614658%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=EvEVi1mhEAbVLK7NQppVJX8i7aqkgCtwbH8GRFr81Fo%3D&reserved=0" target="_blank">http://paste.openstack.org/show/614658/</a>><br>
<br>
Nova database shows correct information on both GPU<br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2F8JS0i6BMitjeBVRJTkRo%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=V%2BFxNgTY2N3hDU6gK31axnLCf1bvz7B7Lw%2FmqY%2BrhT8%3D&reserved=0" target="_blank">
http://paste.openstack.org/show/8JS0i6BMitjeBVRJTkRo/</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2F8JS0i6BMitjeBVRJTkRo%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=V%2BFxNgTY2N3hDU6gK31axnLCf1bvz7B7Lw%2FmqY%2BrhT8%3D&reserved=0" target="_blank">http://paste.openstack.org/show/8JS0i6BMitjeBVRJTkRo/</a>><br>
<br>
<br>
<br>
Now remove ID "1002:68c8" from nova.conf and compute node, and<br>
restart services.<br>
<br>
The pci_passthrough_whitelist and pci_alias only keep<br>
"10de:0ff3" GPU info.<br>
<br>
pci_passthrough_whitelist = { "product_id":"0ff3",<br>
"vendor_id":"10de" }<br>
<br>
pci_alias = { "product_id":"0ff3", "vendor_id":"10de",<br>
"device_type":"type-PCI", "name":"k420" }<br>
<br>
<br>
nova-compute log shows resource tracker report node only have<br>
"10de:0ff3" PCI resource<br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2FVjLinsipne5nM8o0TYcJ%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=GmWsrHuv1DphNJXSKDils8iUWn%2BKbeihjmfDQHQHOMY%3D&reserved=0" target="_blank">
http://paste.openstack.org/show/VjLinsipne5nM8o0TYcJ/</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2FVjLinsipne5nM8o0TYcJ%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=GmWsrHuv1DphNJXSKDils8iUWn%2BKbeihjmfDQHQHOMY%3D&reserved=0" target="_blank">http://paste.openstack.org/show/VjLinsipne5nM8o0TYcJ/</a>><br>
<br>
But in Nova database, "1002:68c8" still exist, and stayed in<br>
"Available" status. Even "deleted" value shows not zero.<br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2FSnJ8AzJYD6wCo7jslIc2%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=9bzrGFdYVtAtwKdTu0ZaxegUah3ZTBbNqAGjCrsT9lk%3D&reserved=0" target="_blank">
http://paste.openstack.org/show/SnJ8AzJYD6wCo7jslIc2/</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org%2Fshow%2FSnJ8AzJYD6wCo7jslIc2%2F&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=9bzrGFdYVtAtwKdTu0ZaxegUah3ZTBbNqAGjCrsT9lk%3D&reserved=0" target="_blank">http://paste.openstack.org/show/SnJ8AzJYD6wCo7jslIc2/</a>><br>
<br>
<br>
Many thanks,<br>
Eddie.<br>
<br>
2017-07-07 9:05 GMT+08:00 Eddie Yen <<a href="mailto:missile0407@gmail.com" target="_blank">missile0407@gmail.com</a><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <mailto:<a href="mailto:missile0407@gmail.com" target="_blank">missile0407@gmail.com</a>>>:<br>
<br>
Uh wait,<br>
<br>
Is that possible it still shows available if PCI device<br>
still exist in the same address?<br>
<br>
Because when I remove the GPU card, I replace it to a SFP+<br>
network card in the same slot.<br>
So when I type lspci the SFP+ card stay in the same address.<br>
<br>
But it still doesn't make any sense because these two cards<br>
definitely not a same VID:PID.<br>
And I set the information as VID:PID in nova.conf<br>
<br>
<br>
I'll try reproduce this issue and put a log on this list.<br>
<br>
Thanks,<br>
<br>
2017-07-07 9:01 GMT+08:00 Jay Pipes <<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><br>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>>:<br>
<br>
Hmm, very odd indeed. Any way you can save the<br>
nova-compute logs from when you removed the GPU and<br>
restarted the nova-compute service and paste those logs<br>
to <a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=%2B6nouVdZuiGwaywLl%2BYGqbqDIbZZIjagLykv6%2BEYrf8%3D&reserved=0" target="_blank">
paste.openstack.org</a> <<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.openstack.org&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=%2B6nouVdZuiGwaywLl%2BYGqbqDIbZZIjagLykv6%2BEYrf8%3D&reserved=0" target="_blank">http://paste.openstack.org</a>>?<br>
Would be useful in tracking down this buggy behaviour...<br>
<br>
Best,<br>
-jay<br>
<br>
On 07/06/2017 08:54 PM, Eddie Yen wrote:<br>
<br>
Hi Jay,<br>
<br>
The status of the "removed" GPU still shows as<br>
"Available" in pci_devices table.<br>
<br>
2017-07-07 8:34 GMT+08:00 Jay Pipes<br>
<<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a> <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>><br>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>>>:<br>
<br>
<br>
Hi again, Eddie :) Answer inline...<br>
<br>
On 07/06/2017 08:14 PM, Eddie Yen wrote:<br>
<br>
Hi everyone,<br>
<br>
I'm using OpenStack Mitaka version<br>
(deployed from Fuel 9.2)<br>
<br>
In present, I installed two different model<br>
of GPU card.<br>
<br>
And wrote these information into pci_alias and<br>
pci_passthrough_whitelist in nova.conf on<br>
Controller and Compute<br>
(the node which installed GPU).<br>
Then restart nova-api, nova-scheduler,and<br>
nova-compute.<br>
<br>
When I check database, both of GPU info<br>
registered in<br>
pci_devices table.<br>
<br>
Now I removed one of the GPU from compute<br>
node, and remove the<br>
information from nova.conf, then restart<br>
services.<br>
<br>
But I check database again, the information<br>
of the removed card<br>
still exist in pci_devices table.<br>
<br>
How can I do to fix this problem?<br>
<br>
<br>
So, when you removed the GPU from the compute<br>
node and restarted the<br>
nova-compute service, it *should* have noticed<br>
you had removed the<br>
GPU and marked that PCI device as deleted. At<br>
least, according to<br>
this code in the PCI manager:<br>
<br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fnova%2Fblob%2Fmaster%2Fnova%2Fpci%2Fmanager.py%23L168-L183&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=CYb%2Fec5fiAkU9LfJ7W6eMxXsS%2F2VpdfaVYSAdcGRy94%3D&reserved=0" target="_blank">
https://github.com/openstack/nova/blob/master/nova/pci/manager.py#L168-L183</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fnova%2Fblob%2Fmaster%2Fnova%2Fpci%2Fmanager.py%23L168-L183&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=CYb%2Fec5fiAkU9LfJ7W6eMxXsS%2F2VpdfaVYSAdcGRy94%3D&reserved=0" target="_blank">https://github.com/openstack/nova/blob/master/nova/pci/manager.py#L168-L183</a>><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fnova%2Fblob%2Fmaster%2Fnova%2Fpci%2Fmanager.py%23L168-L183&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=CYb%2Fec5fiAkU9LfJ7W6eMxXsS%2F2VpdfaVYSAdcGRy94%3D&reserved=0" target="_blank">https://github.com/openstack/nova/blob/master/nova/pci/manager.py#L168-L183</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fnova%2Fblob%2Fmaster%2Fnova%2Fpci%2Fmanager.py%23L168-L183&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=CYb%2Fec5fiAkU9LfJ7W6eMxXsS%2F2VpdfaVYSAdcGRy94%3D&reserved=0" target="_blank">https://github.com/openstack/nova/blob/master/nova/pci/manager.py#L168-L183</a>>><br>
<br>
Question for you: what is the value of the<br>
status field in the<br>
pci_devices table for the GPU that you removed?<br>
<br>
Best,<br>
-jay<br>
<br>
p.s. If you really want to get rid of that<br>
device, simply remove<br>
that record from the pci_devices table. But,<br>
again, it *should* be<br>
removed automatically...<br>
<br>
_______________________________________________<br>
Mailing list:<br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openstack.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopenstack&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=ZpzNaE0Wra4KGRWcluDSyq9lIWTjcOa%2F0uEzllZ6ofI%3D&reserved=0" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openstack.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopenstack&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098583083&sdata=ZpzNaE0Wra4KGRWcluDSyq9lIWTjcOa%2F0uEzllZ6ofI%3D&reserved=0" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openstack.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopenstack&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098593092&sdata=EM1gsCu55xLMlaPGl5QumwnCR%2FEfgNEEF3GpXOCDshE%3D&reserved=0" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openstack.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopenstack&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098593092&sdata=EM1gsCu55xLMlaPGl5QumwnCR%2FEfgNEEF3GpXOCDshE%3D&reserved=0" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>>><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">
openstack@lists.openstack.org</a><br>
<mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
<mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>>><br>
Unsubscribe :<br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openstack.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopenstack&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098593092&sdata=EM1gsCu55xLMlaPGl5QumwnCR%2FEfgNEEF3GpXOCDshE%3D&reserved=0" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openstack.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopenstack&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098593092&sdata=EM1gsCu55xLMlaPGl5QumwnCR%2FEfgNEEF3GpXOCDshE%3D&reserved=0" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openstack.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopenstack&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098593092&sdata=EM1gsCu55xLMlaPGl5QumwnCR%2FEfgNEEF3GpXOCDshE%3D&reserved=0" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openstack.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopenstack&data=02%7C01%7Cmoshele%40mellanox.com%7C21206586310a435b1ddf08d4c7f436df%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636353299098593092&sdata=EM1gsCu55xLMlaPGl5QumwnCR%2FEfgNEEF3GpXOCDshE%3D&reserved=0" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>>><br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</blockquote>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>