[VICTORIA][IRONIC] - Can inspect but not deploy
Hi everyone! I'm dealing with a strange issue today, we deployed IRONIC on a VICTORIA platform, we activated the dnsmasq pxe filtering option at the inspector level, it works great as only IRONIC listed hosts are then served by the dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir. BUT, I'm having a weird issue now. All my nodes are able to get an IP from the DHCP at the INSPECTION step, however, as soon as the inspection step is finished, the mac related file is once again filled with "<mac>,ignore" which prohibits further operations. This means as soon as I put that node as available and try to deploy an "instance" on it (provision the host), it doesn't work as dnsmasq reject the host boot DHCP requests. So, is there a way to instruct the ironic-conductor to edit/allow the host the way the inspector is able to manipulate this file? Did I missed something? Thanks a lot!
Hello Gaël, Which network_interface are you using for your nodes? Is your provisioning network different from the inspection network? Pierre On Mon, 13 Jun 2022 at 10:36, Gaël THEROND <gael.therond@bitswalk.com> wrote:
Hi everyone!
I'm dealing with a strange issue today, we deployed IRONIC on a VICTORIA platform, we activated the dnsmasq pxe filtering option at the inspector level, it works great as only IRONIC listed hosts are then served by the dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir.
BUT, I'm having a weird issue now.
All my nodes are able to get an IP from the DHCP at the INSPECTION step, however, as soon as the inspection step is finished, the mac related file is once again filled with "<mac>,ignore" which prohibits further operations.
This means as soon as I put that node as available and try to deploy an "instance" on it (provision the host), it doesn't work as dnsmasq reject the host boot DHCP requests.
So, is there a way to instruct the ironic-conductor to edit/allow the host the way the inspector is able to manipulate this file?
Did I missed something?
Thanks a lot!
Hi pierre, I’m using a dedicated interface but this interface is the same for all ironic networks inspection/provisioning/cleaning. This interface works fine for inspection, my only issue is the pxe_filter that the ironic inspector process allow during inspection correctly but then tag as disallowed again at the end of the inspection, shouldn’t the deploy process allow the Mac again before booting the node? I can correctly see the conductor instruct the node to boot up using pxe from the kvm console but the BootP process doesn’t load the IPA kernel/initramfs as the dnsmasq pxe discard the request (because of the mac being still tagged as «,ignore » within the hostdir file). I’m a bit disappointed by this behavior. Thanks a lot! Le lun. 13 juin 2022 à 11:41, Pierre Riteau <pierre@stackhpc.com> a écrit :
Hello Gaël,
Which network_interface are you using for your nodes? Is your provisioning network different from the inspection network?
Pierre
On Mon, 13 Jun 2022 at 10:36, Gaël THEROND <gael.therond@bitswalk.com> wrote:
Hi everyone!
I'm dealing with a strange issue today, we deployed IRONIC on a VICTORIA platform, we activated the dnsmasq pxe filtering option at the inspector level, it works great as only IRONIC listed hosts are then served by the dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir.
BUT, I'm having a weird issue now.
All my nodes are able to get an IP from the DHCP at the INSPECTION step, however, as soon as the inspection step is finished, the mac related file is once again filled with "<mac>,ignore" which prohibits further operations.
This means as soon as I put that node as available and try to deploy an "instance" on it (provision the host), it doesn't work as dnsmasq reject the host boot DHCP requests.
So, is there a way to instruct the ironic-conductor to edit/allow the host the way the inspector is able to manipulate this file?
Did I missed something?
Thanks a lot!
Hi Gaël, I am not talking about the physical network interface, but about the `network_interface` field on on your Ironic nodes: https://docs.openstack.org/ironic/latest/admin/multitenancy.html#network-int... Pierre On Mon, 13 Jun 2022 at 12:41, Gaël THEROND <gael.therond@bitswalk.com> wrote:
Hi pierre,
I’m using a dedicated interface but this interface is the same for all ironic networks inspection/provisioning/cleaning.
This interface works fine for inspection, my only issue is the pxe_filter that the ironic inspector process allow during inspection correctly but then tag as disallowed again at the end of the inspection, shouldn’t the deploy process allow the Mac again before booting the node?
I can correctly see the conductor instruct the node to boot up using pxe from the kvm console but the BootP process doesn’t load the IPA kernel/initramfs as the dnsmasq pxe discard the request (because of the mac being still tagged as «,ignore » within the hostdir file).
I’m a bit disappointed by this behavior.
Thanks a lot!
Le lun. 13 juin 2022 à 11:41, Pierre Riteau <pierre@stackhpc.com> a écrit :
Hello Gaël,
Which network_interface are you using for your nodes? Is your provisioning network different from the inspection network?
Pierre
On Mon, 13 Jun 2022 at 10:36, Gaël THEROND <gael.therond@bitswalk.com> wrote:
Hi everyone!
I'm dealing with a strange issue today, we deployed IRONIC on a VICTORIA platform, we activated the dnsmasq pxe filtering option at the inspector level, it works great as only IRONIC listed hosts are then served by the dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir.
BUT, I'm having a weird issue now.
All my nodes are able to get an IP from the DHCP at the INSPECTION step, however, as soon as the inspection step is finished, the mac related file is once again filled with "<mac>,ignore" which prohibits further operations.
This means as soon as I put that node as available and try to deploy an "instance" on it (provision the host), it doesn't work as dnsmasq reject the host boot DHCP requests.
So, is there a way to instruct the ironic-conductor to edit/allow the host the way the inspector is able to manipulate this file?
Did I missed something?
Thanks a lot!
Aaaah yes, sorry my bad, as we do use Kolla-ansible I forget about this one, so we're using Flat network interface. Le lun. 13 juin 2022 à 13:14, Pierre Riteau <pierre@stackhpc.com> a écrit :
Hi Gaël,
I am not talking about the physical network interface, but about the `network_interface` field on on your Ironic nodes: https://docs.openstack.org/ironic/latest/admin/multitenancy.html#network-int...
Pierre
On Mon, 13 Jun 2022 at 12:41, Gaël THEROND <gael.therond@bitswalk.com> wrote:
Hi pierre,
I’m using a dedicated interface but this interface is the same for all ironic networks inspection/provisioning/cleaning.
This interface works fine for inspection, my only issue is the pxe_filter that the ironic inspector process allow during inspection correctly but then tag as disallowed again at the end of the inspection, shouldn’t the deploy process allow the Mac again before booting the node?
I can correctly see the conductor instruct the node to boot up using pxe from the kvm console but the BootP process doesn’t load the IPA kernel/initramfs as the dnsmasq pxe discard the request (because of the mac being still tagged as «,ignore » within the hostdir file).
I’m a bit disappointed by this behavior.
Thanks a lot!
Le lun. 13 juin 2022 à 11:41, Pierre Riteau <pierre@stackhpc.com> a écrit :
Hello Gaël,
Which network_interface are you using for your nodes? Is your provisioning network different from the inspection network?
Pierre
On Mon, 13 Jun 2022 at 10:36, Gaël THEROND <gael.therond@bitswalk.com> wrote:
Hi everyone!
I'm dealing with a strange issue today, we deployed IRONIC on a VICTORIA platform, we activated the dnsmasq pxe filtering option at the inspector level, it works great as only IRONIC listed hosts are then served by the dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir.
BUT, I'm having a weird issue now.
All my nodes are able to get an IP from the DHCP at the INSPECTION step, however, as soon as the inspection step is finished, the mac related file is once again filled with "<mac>,ignore" which prohibits further operations.
This means as soon as I put that node as available and try to deploy an "instance" on it (provision the host), it doesn't work as dnsmasq reject the host boot DHCP requests.
So, is there a way to instruct the ironic-conductor to edit/allow the host the way the inspector is able to manipulate this file?
Did I missed something?
Thanks a lot!
Alright, I just tested to disable the pxe_filter option and it works, the host is then deployed using the glance image and neutron network settings correctly. I think there is either a bug or maybe something that I didn’t catch up but it’s pretty annoying. I’ll propose a patch to kolla-ansible as well as I discovered that you can’t disable this feature for now due to the template shape. Any ideas about why the conductor isn’t able to whitelist this node ? Thanks a lot! Le lun. 13 juin 2022 à 13:43, Gaël THEROND <gael.therond@bitswalk.com> a écrit :
Aaaah yes, sorry my bad, as we do use Kolla-ansible I forget about this one, so we're using Flat network interface.
Le lun. 13 juin 2022 à 13:14, Pierre Riteau <pierre@stackhpc.com> a écrit :
Hi Gaël,
I am not talking about the physical network interface, but about the `network_interface` field on on your Ironic nodes: https://docs.openstack.org/ironic/latest/admin/multitenancy.html#network-int...
Pierre
On Mon, 13 Jun 2022 at 12:41, Gaël THEROND <gael.therond@bitswalk.com> wrote:
Hi pierre,
I’m using a dedicated interface but this interface is the same for all ironic networks inspection/provisioning/cleaning.
This interface works fine for inspection, my only issue is the pxe_filter that the ironic inspector process allow during inspection correctly but then tag as disallowed again at the end of the inspection, shouldn’t the deploy process allow the Mac again before booting the node?
I can correctly see the conductor instruct the node to boot up using pxe from the kvm console but the BootP process doesn’t load the IPA kernel/initramfs as the dnsmasq pxe discard the request (because of the mac being still tagged as «,ignore » within the hostdir file).
I’m a bit disappointed by this behavior.
Thanks a lot!
Le lun. 13 juin 2022 à 11:41, Pierre Riteau <pierre@stackhpc.com> a écrit :
Hello Gaël,
Which network_interface are you using for your nodes? Is your provisioning network different from the inspection network?
Pierre
On Mon, 13 Jun 2022 at 10:36, Gaël THEROND <gael.therond@bitswalk.com> wrote:
Hi everyone!
I'm dealing with a strange issue today, we deployed IRONIC on a VICTORIA platform, we activated the dnsmasq pxe filtering option at the inspector level, it works great as only IRONIC listed hosts are then served by the dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir.
BUT, I'm having a weird issue now.
All my nodes are able to get an IP from the DHCP at the INSPECTION step, however, as soon as the inspection step is finished, the mac related file is once again filled with "<mac>,ignore" which prohibits further operations.
This means as soon as I put that node as available and try to deploy an "instance" on it (provision the host), it doesn't work as dnsmasq reject the host boot DHCP requests.
So, is there a way to instruct the ironic-conductor to edit/allow the host the way the inspector is able to manipulate this file?
Did I missed something?
Thanks a lot!
On 6/13/22 12:41, Gaël THEROND wrote:
Hi pierre,
I’m using a dedicated interface but this interface is the same for all ironic networks inspection/provisioning/cleaning.
This interface works fine for inspection, my only issue is the pxe_filter that the ironic inspector process allow during inspection correctly but then tag as disallowed again at the end of the inspection, shouldn’t the deploy process allow the Mac again before booting the node?
The ironic dnsmasq pxe filter is only supposed to allow DHCP requests when a node is inspected. There are two DHCP services, one for inspector and the other one is typically hosted by neutron for provisioning/cleaning/rescue. Only the dnsmasq for inspector uses the hostdir with ',ignore' files.
I can correctly see the conductor instruct the node to boot up using pxe from the kvm console but the BootP process doesn’t load the IPA kernel/initramfs as the dnsmasq pxe discard the request (because of the mac being still tagged as «,ignore » within the hostdir file).
When a node is provisioned/cleaned/rescued the DHCP instance in neutron should be the one providing the DHCP service, not the dnsmasq instance for inspector service. i.e the ',ignore' entries should remain as 'ignore' to ensure the node does not get a DHCP reply from inspector's DHCP service which would deliver the wrong DHCP options.
I’m a bit disappointed by this behavior.
Thanks a lot!
Le lun. 13 juin 2022 à 11:41, Pierre Riteau <pierre@stackhpc.com <mailto:pierre@stackhpc.com>> a écrit :
Hello Gaël,
Which network_interface are you using for your nodes? Is your provisioning network different from the inspection network?
Pierre
On Mon, 13 Jun 2022 at 10:36, Gaël THEROND <gael.therond@bitswalk.com <mailto:gael.therond@bitswalk.com>> wrote:
Hi everyone!
I'm dealing with a strange issue today, we deployed IRONIC on a VICTORIA platform, we activated the dnsmasq pxe filtering option at the inspector level, it works great as only IRONIC listed hosts are then served by the dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir.
BUT, I'm having a weird issue now.
All my nodes are able to get an IP from the DHCP at the INSPECTION step, however, as soon as the inspection step is finished, the mac related file is once again filled with "<mac>,ignore" which prohibits further operations.
This means as soon as I put that node as available and try to deploy an "instance" on it (provision the host), it doesn't work as dnsmasq reject the host boot DHCP requests.
So, is there a way to instruct the ironic-conductor to edit/allow the host the way the inspector is able to manipulate this file?
Did I missed something?
Thanks a lot!
participants (3)
-
Gaël THEROND
-
Harald Jensas
-
Pierre Riteau