[Kolla-Ansible] [Ironic] Node Deploy Failed with MissingAuthPlugin
Hi Kolla and Ironic folks,
While playing with stable/kolla release to fix the No VIF found error. Looks like adding Readers role to ironic user fixed it. But run into Missing Auth plugin [2] issue while image is being deployed to ironic node. Openstack endpoint list command [1] shows the correct IP of the controller.
My question is:
1. Kolla-Ansible needs explicite Ironic keystone configuration in Ironic.conf? Following this link https://docs.openstack.org/kolla-ansible/yoga/reference/bare-metal/ironic-guide.html which does not seems to need any keystone configurations. This issue is observed when
1. --boot-interface redfish-virtual-media 2. --deploy-interface ramdisk 3. --driver-info deploy_iso= 4. -instance-info kernel & ramdisk
2. My passwords.yaml has
1. ironic_database_password: 2. ironic_inspector_database_password: 3. ironic_inspector_keystone_password: 4. ironic_keystone_password:
Any suggestions please.
Thank you.
Ali
[1] openstack endpoint list | grep ironic | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | internal | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | public | http://<IP>:6385 | | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | public | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | internal | http://<IP>:6385
[2] *Error: *Detected change in error condition on node Node1. Deploy step deploy.deploy failed with MissingAuthPlugin: An auth plugin is required to determine endpoint URL.. An unhandled exception was encountered while aborting. More information may be found in the log file.
Greetings Ali,
I'm really glad to hear the reader role fixed things up for you. That really sounds like a bug or a defect since at that point with Kolla Yoga release. My understanding is that when an admin user was created as part keystone deployment, the "member" and "reader" permission roles were also automatically added. It might be kolla did something different or hadn't adopted the newer RBAC model yet.. If you haven't already done so, double check that your user *also* has the "member" role. Ironic's access model is graduated by access level as a result of the community Secure RBAC effort.
Anyhow, for this issue you're encountering, I highly suspect this is because you're attempting to use the redfish-virtual-media boot interface driver, which was originally modeled to upload artifacts to swift, but it can also be set to just store the artifacts locally for a self-hosted http server.
My guess is that your Kolla configuration lacks swift and the associated object-store endpoint configuration.
You can test my theory and solution by going to ironic.conf's "[redfish]" configuration section, and try setting the "use_swift" setting to "false". If you then send the ironic-conductor a HUP signal, it will reload the configuration and your next deploy attempt will likely get further.
Have a great day, and let us know!
-Julia
On Sun, Sep 29, 2024 at 12:38 AM Ali Hussain alihussain448@gmail.com wrote:
Hi Kolla and Ironic folks,
While playing with stable/kolla release to fix the No VIF found error. Looks like adding Readers role to ironic user fixed it. But run into Missing Auth plugin [2] issue while image is being deployed to ironic node. Openstack endpoint list command [1] shows the correct IP of the controller.
My question is:
- Kolla-Ansible needs explicite Ironic keystone configuration in
Ironic.conf? Following this link https://docs.openstack.org/kolla-ansible/yoga/reference/bare-metal/ironic-guide.html which does not seems to need any keystone configurations. This issue is observed when
--boot-interface redfish-virtual-media
--deploy-interface ramdisk
--driver-info deploy_iso=
-instance-info kernel & ramdisk
My passwords.yaml has
- ironic_database_password:
- ironic_inspector_database_password:
- ironic_inspector_keystone_password:
- ironic_keystone_password:
Any suggestions please.
Thank you.
Ali
[1] openstack endpoint list | grep ironic | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | internal | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | public | http://<IP>:6385 | | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | public | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | internal | http://<IP>:6385
[2] *Error: *Detected change in error condition on node Node1. Deploy step deploy.deploy failed with MissingAuthPlugin: An auth plugin is required to determine endpoint URL.. An unhandled exception was encountered while aborting. More information may be found in the log file.
Greetings Julia,
Appreciate your help. Here is what I tried and the results with Redfish driver loading media over http.
1. After applying the "[redfish]" configuration section, and setting the "use_swift" setting to "false". The missing auth plugin is not appearing but No VIF found error is thrown again at the deploy step. Tried to set VIF ID using botb of the following commands,
openstack baremetal node vif attach <NodeID> <VID dhcp port ID from network> openstack baremetal port set <Node port ID> --extra vif_port_id= <VID dhcp port ID from network>
2. If I use the ipxe and ramdisk mode of deployment the NO VIF error again show up.
What could be the issue in this scenario now?
Have a nice day.
Ali
On Sun, Sep 29, 2024 at 8:55 AM Julia Kreger juliaashleykreger@gmail.com wrote:
Greetings Ali,
I'm really glad to hear the reader role fixed things up for you. That really sounds like a bug or a defect since at that point with Kolla Yoga release. My understanding is that when an admin user was created as part keystone deployment, the "member" and "reader" permission roles were also automatically added. It might be kolla did something different or hadn't adopted the newer RBAC model yet.. If you haven't already done so, double check that your user *also* has the "member" role. Ironic's access model is graduated by access level as a result of the community Secure RBAC effort.
Anyhow, for this issue you're encountering, I highly suspect this is because you're attempting to use the redfish-virtual-media boot interface driver, which was originally modeled to upload artifacts to swift, but it can also be set to just store the artifacts locally for a self-hosted http server.
My guess is that your Kolla configuration lacks swift and the associated object-store endpoint configuration.
You can test my theory and solution by going to ironic.conf's "[redfish]" configuration section, and try setting the "use_swift" setting to "false". If you then send the ironic-conductor a HUP signal, it will reload the configuration and your next deploy attempt will likely get further.
Have a great day, and let us know!
-Julia
On Sun, Sep 29, 2024 at 12:38 AM Ali Hussain alihussain448@gmail.com wrote:
Hi Kolla and Ironic folks,
While playing with stable/kolla release to fix the No VIF found error. Looks like adding Readers role to ironic user fixed it. But run into Missing Auth plugin [2] issue while image is being deployed to ironic node. Openstack endpoint list command [1] shows the correct IP of the controller.
My question is:
- Kolla-Ansible needs explicite Ironic keystone configuration in
Ironic.conf? Following this link https://docs.openstack.org/kolla-ansible/yoga/reference/bare-metal/ironic-guide.html which does not seems to need any keystone configurations. This issue is observed when
--boot-interface redfish-virtual-media
--deploy-interface ramdisk
--driver-info deploy_iso=
-instance-info kernel & ramdisk
My passwords.yaml has
- ironic_database_password:
- ironic_inspector_database_password:
- ironic_inspector_keystone_password:
- ironic_keystone_password:
Any suggestions please.
Thank you.
Ali
[1] openstack endpoint list | grep ironic | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | internal | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | public | http://<IP>:6385 | | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | public | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | internal | http://<IP>:6385
[2] *Error: *Detected change in error condition on node Node1. Deploy step deploy.deploy failed with MissingAuthPlugin: An auth plugin is required to determine endpoint URL.. An unhandled exception was encountered while aborting. More information may be found in the log file.
On Mon, Sep 30, 2024 at 11:48 AM Ali Hussain alihussain448@gmail.com wrote:
Greetings Julia,
Appreciate your help. Here is what I tried and the results with Redfish driver loading media over http.
- After applying the "[redfish]" configuration section, and setting the
"use_swift" setting to "false". The missing auth plugin is not appearing but No VIF found error is thrown again at the deploy step. Tried to set VIF ID using botb of the following commands,
openstack baremetal node vif attach <NodeID> <VID dhcp port ID from network>
You mean you're using the neutron port ID? It sounds like it might be another RBAC issue, but this time in Neutron?
openstack baremetal port set <Node port ID> --extra vif_port_id= <VID dhcp port ID from network>
This way is functionally the same as vif attachment, however has been deprecated for years. I'm actually not even sure if we actually support it.
- If I use the ipxe and ramdisk mode of deployment the NO VIF error again
show up.
Yes, the vif error will always appear if Ironic cannot see and send binding profile information to Neutron to complete the network configuration.
What could be the issue in this scenario now?
Have a nice day.
Ali
On Sun, Sep 29, 2024 at 8:55 AM Julia Kreger juliaashleykreger@gmail.com wrote:
Greetings Ali,
I'm really glad to hear the reader role fixed things up for you. That really sounds like a bug or a defect since at that point with Kolla Yoga release. My understanding is that when an admin user was created as part keystone deployment, the "member" and "reader" permission roles were also automatically added. It might be kolla did something different or hadn't adopted the newer RBAC model yet.. If you haven't already done so, double check that your user *also* has the "member" role. Ironic's access model is graduated by access level as a result of the community Secure RBAC effort.
Anyhow, for this issue you're encountering, I highly suspect this is because you're attempting to use the redfish-virtual-media boot interface driver, which was originally modeled to upload artifacts to swift, but it can also be set to just store the artifacts locally for a self-hosted http server.
My guess is that your Kolla configuration lacks swift and the associated object-store endpoint configuration.
You can test my theory and solution by going to ironic.conf's "[redfish]" configuration section, and try setting the "use_swift" setting to "false". If you then send the ironic-conductor a HUP signal, it will reload the configuration and your next deploy attempt will likely get further.
Have a great day, and let us know!
-Julia
On Sun, Sep 29, 2024 at 12:38 AM Ali Hussain alihussain448@gmail.com wrote:
Hi Kolla and Ironic folks,
While playing with stable/kolla release to fix the No VIF found error. Looks like adding Readers role to ironic user fixed it. But run into Missing Auth plugin [2] issue while image is being deployed to ironic node. Openstack endpoint list command [1] shows the correct IP of the controller.
My question is:
- Kolla-Ansible needs explicite Ironic keystone configuration in
Ironic.conf? Following this link https://docs.openstack.org/kolla-ansible/yoga/reference/bare-metal/ironic-guide.html which does not seems to need any keystone configurations. This issue is observed when
--boot-interface redfish-virtual-media
--deploy-interface ramdisk
--driver-info deploy_iso=
-instance-info kernel & ramdisk
My passwords.yaml has
- ironic_database_password:
- ironic_inspector_database_password:
- ironic_inspector_keystone_password:
- ironic_keystone_password:
Any suggestions please.
Thank you.
Ali
[1] openstack endpoint list | grep ironic | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | internal | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | public | http://<IP>:6385 | | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | public | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | internal | http://<IP>:6385
[2] *Error: *Detected change in error condition on node Node1. Deploy step deploy.deploy failed with MissingAuthPlugin: An auth plugin is required to determine endpoint URL.. An unhandled exception was encountered while aborting. More information may be found in the log file.
Yes, trying to link Neutron port to Ironic node. Neutron user role also need some fix?
Ali
On Tue, Oct 1, 2024 at 10:32 AM Julia Kreger juliaashleykreger@gmail.com wrote:
On Mon, Sep 30, 2024 at 11:48 AM Ali Hussain alihussain448@gmail.com wrote:
Greetings Julia,
Appreciate your help. Here is what I tried and the results with Redfish driver loading media over http.
- After applying the "[redfish]" configuration section, and setting
the "use_swift" setting to "false". The missing auth plugin is not appearing but No VIF found error is thrown again at the deploy step. Tried to set VIF ID using botb of the following commands,
openstack baremetal node vif attach <NodeID> <VID dhcp port ID from network>
You mean you're using the neutron port ID? It sounds like it might be another RBAC issue, but this time in Neutron?
openstack baremetal port set <Node port ID> --extra vif_port_id= <VID dhcp port ID from network>
This way is functionally the same as vif attachment, however has been deprecated for years. I'm actually not even sure if we actually support it.
- If I use the ipxe and ramdisk mode of deployment the NO VIF error
again show up.
Yes, the vif error will always appear if Ironic cannot see and send binding profile information to Neutron to complete the network configuration.
What could be the issue in this scenario now?
Have a nice day.
Ali
On Sun, Sep 29, 2024 at 8:55 AM Julia Kreger juliaashleykreger@gmail.com wrote:
Greetings Ali,
I'm really glad to hear the reader role fixed things up for you. That really sounds like a bug or a defect since at that point with Kolla Yoga release. My understanding is that when an admin user was created as part keystone deployment, the "member" and "reader" permission roles were also automatically added. It might be kolla did something different or hadn't adopted the newer RBAC model yet.. If you haven't already done so, double check that your user *also* has the "member" role. Ironic's access model is graduated by access level as a result of the community Secure RBAC effort.
Anyhow, for this issue you're encountering, I highly suspect this is because you're attempting to use the redfish-virtual-media boot interface driver, which was originally modeled to upload artifacts to swift, but it can also be set to just store the artifacts locally for a self-hosted http server.
My guess is that your Kolla configuration lacks swift and the associated object-store endpoint configuration.
You can test my theory and solution by going to ironic.conf's "[redfish]" configuration section, and try setting the "use_swift" setting to "false". If you then send the ironic-conductor a HUP signal, it will reload the configuration and your next deploy attempt will likely get further.
Have a great day, and let us know!
-Julia
On Sun, Sep 29, 2024 at 12:38 AM Ali Hussain alihussain448@gmail.com wrote:
Hi Kolla and Ironic folks,
While playing with stable/kolla release to fix the No VIF found error. Looks like adding Readers role to ironic user fixed it. But run into Missing Auth plugin [2] issue while image is being deployed to ironic node. Openstack endpoint list command [1] shows the correct IP of the controller.
My question is:
- Kolla-Ansible needs explicite Ironic keystone configuration in
Ironic.conf? Following this link https://docs.openstack.org/kolla-ansible/yoga/reference/bare-metal/ironic-guide.html which does not seems to need any keystone configurations. This issue is observed when
--boot-interface redfish-virtual-media
--deploy-interface ramdisk
--driver-info deploy_iso=
-instance-info kernel & ramdisk
My passwords.yaml has
- ironic_database_password:
- ironic_inspector_database_password:
- ironic_inspector_keystone_password:
- ironic_keystone_password:
Any suggestions please.
Thank you.
Ali
[1] openstack endpoint list | grep ironic | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | internal | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | public | http://<IP>:6385 | | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | public | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | internal | http://<IP>:6385
[2] *Error: *Detected change in error condition on node Node1. Deploy step deploy.deploy failed with MissingAuthPlugin: An auth plugin is required to determine endpoint URL.. An unhandled exception was encountered while aborting. More information may be found in the log file.
The user ironic uses to access Neutron.
On Tue, Oct 1, 2024 at 9:46 AM Ali Hussain alihussain448@gmail.com wrote:
Yes, trying to link Neutron port to Ironic node. Neutron user role also need some fix?
Ali
On Tue, Oct 1, 2024 at 10:32 AM Julia Kreger juliaashleykreger@gmail.com wrote:
On Mon, Sep 30, 2024 at 11:48 AM Ali Hussain alihussain448@gmail.com wrote:
Greetings Julia,
Appreciate your help. Here is what I tried and the results with Redfish driver loading media over http.
- After applying the "[redfish]" configuration section, and setting
the "use_swift" setting to "false". The missing auth plugin is not appearing but No VIF found error is thrown again at the deploy step. Tried to set VIF ID using botb of the following commands,
openstack baremetal node vif attach <NodeID> <VID dhcp port ID from network>
You mean you're using the neutron port ID? It sounds like it might be another RBAC issue, but this time in Neutron?
openstack baremetal port set <Node port ID> --extra vif_port_id= <VID dhcp port ID from network>
This way is functionally the same as vif attachment, however has been deprecated for years. I'm actually not even sure if we actually support it.
- If I use the ipxe and ramdisk mode of deployment the NO VIF error
again show up.
Yes, the vif error will always appear if Ironic cannot see and send binding profile information to Neutron to complete the network configuration.
What could be the issue in this scenario now?
Have a nice day.
Ali
On Sun, Sep 29, 2024 at 8:55 AM Julia Kreger < juliaashleykreger@gmail.com> wrote:
Greetings Ali,
I'm really glad to hear the reader role fixed things up for you. That really sounds like a bug or a defect since at that point with Kolla Yoga release. My understanding is that when an admin user was created as part keystone deployment, the "member" and "reader" permission roles were also automatically added. It might be kolla did something different or hadn't adopted the newer RBAC model yet.. If you haven't already done so, double check that your user *also* has the "member" role. Ironic's access model is graduated by access level as a result of the community Secure RBAC effort.
Anyhow, for this issue you're encountering, I highly suspect this is because you're attempting to use the redfish-virtual-media boot interface driver, which was originally modeled to upload artifacts to swift, but it can also be set to just store the artifacts locally for a self-hosted http server.
My guess is that your Kolla configuration lacks swift and the associated object-store endpoint configuration.
You can test my theory and solution by going to ironic.conf's "[redfish]" configuration section, and try setting the "use_swift" setting to "false". If you then send the ironic-conductor a HUP signal, it will reload the configuration and your next deploy attempt will likely get further.
Have a great day, and let us know!
-Julia
On Sun, Sep 29, 2024 at 12:38 AM Ali Hussain alihussain448@gmail.com wrote:
Hi Kolla and Ironic folks,
While playing with stable/kolla release to fix the No VIF found error. Looks like adding Readers role to ironic user fixed it. But run into Missing Auth plugin [2] issue while image is being deployed to ironic node. Openstack endpoint list command [1] shows the correct IP of the controller.
My question is:
- Kolla-Ansible needs explicite Ironic keystone configuration in
Ironic.conf? Following this link https://docs.openstack.org/kolla-ansible/yoga/reference/bare-metal/ironic-guide.html which does not seems to need any keystone configurations. This issue is observed when
--boot-interface redfish-virtual-media
--deploy-interface ramdisk
--driver-info deploy_iso=
-instance-info kernel & ramdisk
My passwords.yaml has
- ironic_database_password:
- ironic_inspector_database_password:
- ironic_inspector_keystone_password:
- ironic_keystone_password:
Any suggestions please.
Thank you.
Ali
[1] openstack endpoint list | grep ironic | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | internal | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | public | http://<IP>:6385 | | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | public | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | internal | http://<IP>:6385
[2] *Error: *Detected change in error condition on node Node1. Deploy step deploy.deploy failed with MissingAuthPlugin: An auth plugin is required to determine endpoint URL.. An unhandled exception was encountered while aborting. More information may be found in the log file.
Added missing roles to Neutron user. User redfish and http url for the ISO images. Looks like kolla is unable to store them.
Got following exception now.
Error: detected change in error condition on nodr. Failed to prepare to deploy. Exception: Error 18 invalid cross-device link: /var/lib/ironic/master_iso_images/*image_id* -> /tmp/tmpdirr/boot.iso
Ali
On Tue, Oct 1, 2024 at 12:41 PM Julia Kreger juliaashleykreger@gmail.com wrote:
The user ironic uses to access Neutron.
On Tue, Oct 1, 2024 at 9:46 AM Ali Hussain alihussain448@gmail.com wrote:
Yes, trying to link Neutron port to Ironic node. Neutron user role also need some fix?
Ali
On Tue, Oct 1, 2024 at 10:32 AM Julia Kreger juliaashleykreger@gmail.com wrote:
On Mon, Sep 30, 2024 at 11:48 AM Ali Hussain alihussain448@gmail.com wrote:
Greetings Julia,
Appreciate your help. Here is what I tried and the results with Redfish driver loading media over http.
- After applying the "[redfish]" configuration section, and setting
the "use_swift" setting to "false". The missing auth plugin is not appearing but No VIF found error is thrown again at the deploy step. Tried to set VIF ID using botb of the following commands,
openstack baremetal node vif attach <NodeID> <VID dhcp port ID from network>
You mean you're using the neutron port ID? It sounds like it might be another RBAC issue, but this time in Neutron?
openstack baremetal port set <Node port ID> --extra vif_port_id= <VID dhcp port ID from network>
This way is functionally the same as vif attachment, however has been deprecated for years. I'm actually not even sure if we actually support it.
- If I use the ipxe and ramdisk mode of deployment the NO VIF error
again show up.
Yes, the vif error will always appear if Ironic cannot see and send binding profile information to Neutron to complete the network configuration.
What could be the issue in this scenario now?
Have a nice day.
Ali
On Sun, Sep 29, 2024 at 8:55 AM Julia Kreger < juliaashleykreger@gmail.com> wrote:
Greetings Ali,
I'm really glad to hear the reader role fixed things up for you. That really sounds like a bug or a defect since at that point with Kolla Yoga release. My understanding is that when an admin user was created as part keystone deployment, the "member" and "reader" permission roles were also automatically added. It might be kolla did something different or hadn't adopted the newer RBAC model yet.. If you haven't already done so, double check that your user *also* has the "member" role. Ironic's access model is graduated by access level as a result of the community Secure RBAC effort.
Anyhow, for this issue you're encountering, I highly suspect this is because you're attempting to use the redfish-virtual-media boot interface driver, which was originally modeled to upload artifacts to swift, but it can also be set to just store the artifacts locally for a self-hosted http server.
My guess is that your Kolla configuration lacks swift and the associated object-store endpoint configuration.
You can test my theory and solution by going to ironic.conf's "[redfish]" configuration section, and try setting the "use_swift" setting to "false". If you then send the ironic-conductor a HUP signal, it will reload the configuration and your next deploy attempt will likely get further.
Have a great day, and let us know!
-Julia
On Sun, Sep 29, 2024 at 12:38 AM Ali Hussain alihussain448@gmail.com wrote:
Hi Kolla and Ironic folks,
While playing with stable/kolla release to fix the No VIF found error. Looks like adding Readers role to ironic user fixed it. But run into Missing Auth plugin [2] issue while image is being deployed to ironic node. Openstack endpoint list command [1] shows the correct IP of the controller.
My question is:
- Kolla-Ansible needs explicite Ironic keystone configuration in
Ironic.conf? Following this link https://docs.openstack.org/kolla-ansible/yoga/reference/bare-metal/ironic-guide.html which does not seems to need any keystone configurations. This issue is observed when
--boot-interface redfish-virtual-media
--deploy-interface ramdisk
--driver-info deploy_iso=
-instance-info kernel & ramdisk
My passwords.yaml has
- ironic_database_password:
- ironic_inspector_database_password:
- ironic_inspector_keystone_password:
- ironic_keystone_password:
Any suggestions please.
Thank you.
Ali
[1] openstack endpoint list | grep ironic | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | internal | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | public | http://<IP>:6385 | | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | public | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | internal | http://<IP>:6385
[2] *Error: *Detected change in error condition on node Node1. Deploy step deploy.deploy failed with MissingAuthPlugin: An auth plugin is required to determine endpoint URL.. An unhandled exception was encountered while aborting. More information may be found in the log file.
participants (2)
-
Ali Hussain
-
Julia Kreger