On Thu, 2019-01-31 at 11:30 -0500, Michael Turek wrote:
Hello all,
Our ironic job has been broken and it seems to be due to a lack of IPs. We allocate two IPs to our job, one for the dhcp server, and one for the target node. This had been working for as long as the job has existed but recently (since about early December 2018), we've been broken.
The job is able to clean the node during devstack, successfully deploy to the node during the tempest run, and is successfully validated via ssh. The node then moves to clean failed with a network error [1], and the job subsequently fails. Sometime between the validation and attempting to clean, the neutron port associated with the ironic port is deleted and a new port comes into existence. Where I'm having trouble is finding out what this port is. Based on it's MAC address It's a virtual port, and its MAC is not the same as the ironic port.
We could add an IP to the job to fix it, but I'd rather not do that needlessly.
Any insight or advice would be appreciated here!
While working on the neutron events I noticed a pattern I thought was a bit strange. (Note, this was with neutron networking.)
Create nova baremetal instance:
1. The tenant VIF is created. 2. The provision port is created. 3. Provision port plugged (bound) 4. Provision port un-plugged (deleted) 5. Tenant port plugged (bound)
On nova delete of barametal instance:
1. Tenant VIF is un-plugged (unbound) 2. Cleaning port created 3. Cleaning port plugged (bound) 4. Cleaning port un-plugged (deleted) 5. Tenant port deleted
I think step 5, deleting the tenant port could happen after step 1. But it looks like it is'nt deleted before after cleaning is done.
If this is the case with flat networks as well it could explain why you get the error on cleaning. The "tenant" port still exist, and there are no free IP's in the allocation pool to create a new port for cleaning.
-- Harald