[masakari] Compute service with name XXXXX not found.
Hi, I have some problem with masakari. I can create segment (from CLI and Horizon), but can't create host (the same result from Horizon and CLI). openstack segment host create XXXXX COMPUTE SSH segment_id returns BadRequest: Compute service with name XXXXX could not be found. XXXXX is the name which Horizon suggest, and it's a name of compute host. openstack compute service list returns proper list with state up/enabled on compute hosts (zone nova) Maybe I misunderstood some parameters of host create? As "type" I use COMPUTE, what value should it be? From "Binary" column of openstack compute service list? What is "control_attributes" field, because documentation lacks preceise information what value should be there and what is it use for. Tried to found some info on this error but I haven't found anything... Thanks in advance for any help. Best regards Adam Tomas
On Fri, Jun 11, 2021 at 11:49 AM at <bkslash@poczta.onet.pl> wrote:
Hi,
Hello,
I have some problem with masakari. I can create segment (from CLI and Horizon), but can't create host (the same result from Horizon and CLI).
openstack segment host create XXXXX COMPUTE SSH segment_id
returns BadRequest: Compute service with name XXXXX could not be found. XXXXX is the name which Horizon suggest, and it's a name of compute host.
openstack compute service list returns proper list with state up/enabled on compute hosts (zone nova) Maybe I misunderstood some parameters of host create? As "type" I use COMPUTE, what value should it be? From "Binary" column of openstack compute service list? What is "control_attributes" field, because documentation lacks preceise information what value should be there and what is it use for. Tried to found some info on this error but I haven't found anything...
The name should be as it is listed by nova. Masakari is querying nova for that compute host. The exact query can be run using: openstack compute service list --service nova-compute --host $HOSTNAME where $HOSTNAME is the desired hostname. The type should be "COMPUTE" and folks often use "SSH" for control_attributes (but it has no meaning). -yoctozepto
Hi, thx for the answer.
openstack compute service list --service nova-compute --host $HOSTNAME so in
openstack segment host create I should use name which is displayed in "Host" column, right? So that's what I do :( openstack compute service list --service nova-compute ID Binary Host Zone Status State 20 nova-compute XXXXX nova enabled up openstack segment host create XXXXX COMPUTE SSH 00dd5bxxxxxx and still "Compute service with name XXXXX could not be found"..... How masakari discovers hosts? Best regards Adam Tomas
Wiadomość napisana przez Radosław Piliszek <radoslaw.piliszek@gmail.com> w dniu 11.06.2021, o godz. 13:26:
On Fri, Jun 11, 2021 at 11:49 AM at <bkslash@poczta.onet.pl> wrote:
Hi,
Hello,
I have some problem with masakari. I can create segment (from CLI and Horizon), but can't create host (the same result from Horizon and CLI).
openstack segment host create XXXXX COMPUTE SSH segment_id
returns BadRequest: Compute service with name XXXXX could not be found. XXXXX is the name which Horizon suggest, and it's a name of compute host.
openstack compute service list returns proper list with state up/enabled on compute hosts (zone nova) Maybe I misunderstood some parameters of host create? As "type" I use COMPUTE, what value should it be? From "Binary" column of openstack compute service list? What is "control_attributes" field, because documentation lacks preceise information what value should be there and what is it use for. Tried to found some info on this error but I haven't found anything...
The name should be as it is listed by nova. Masakari is querying nova for that compute host. The exact query can be run using: openstack compute service list --service nova-compute --host $HOSTNAME where $HOSTNAME is the desired hostname. The type should be "COMPUTE" and folks often use "SSH" for control_attributes (but it has no meaning).
-yoctozepto
On Fri, Jun 11, 2021 at 2:47 PM at <bkslash@poczta.onet.pl> wrote:
Hi, thx for the answer.
openstack compute service list --service nova-compute --host $HOSTNAME so in
openstack segment host create
I should use name which is displayed in "Host" column, right? So that's what I do :(
Yes.
openstack compute service list --service nova-compute
ID Binary Host Zone Status State 20 nova-compute XXXXX nova enabled up
openstack segment host create XXXXX COMPUTE SSH 00dd5bxxxxxx
and still "Compute service with name XXXXX could not be found".....
How masakari discovers hosts?
I wrote this already: openstack compute service list --service nova-compute --host $HOSTNAME did you try including the same hostname in this command? If it works and Masakari does not, I would make sure you set up Masakari to speak to the right Nova API. Finally, if all else fails, please paste (e.g. https://paste.ubuntu.com/ ) masakari api logs for those rejected host creations. Though do that with debug=True in the config [DEFAULT] section. -yoctozepto
openstack compute service list --service nova-compute --host $HOSTNAME did you try including the same hostname in this command? yes, and it returns the same as "openstack compute service list" but of course only for host XXXXX
If it works and Masakari does not, I would make sure you set up Masakari to speak to the right Nova API. I'm using kolla-ansible, all masakari configuration was generated based on globals.yaml and inventory file while deployment, so it should work almost "out of the box". Does masakari speak to nova via RabbitMQ? How else can I check which port/IP masakari speaks to? In logs I can only see requests TO masakari API, not where masakari tries to check hypervisor...
Though do that with debug=True in the config [DEFAULT] section. not much in logs, even with debug enabled....
2021-06-11 14:45:49.111 959 DEBUG masakari.compute.nova [req-e9a58522-858d-4025-9c43-f9fee744a0db nova - - - -] Creating a Nova client using "nova" user novaclient /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/compute/nova.py:102 2021-06-11 14:45:49.232 959 INFO masakari.compute.nova [req-e9a58522-858d-4025-9c43-f9fee744a0db nova - - - -] Call compute service find command to get list of matching hypervisor name 'XXXXX' 2021-06-11 14:45:49.829 959 INFO masakari.api.openstack.wsgi [req-e9a58522-858d-4025-9c43-f9fee744a0db nova - - - -] HTTP exception thrown: Compute service with name XXXXX could not be found. 2021-06-11 14:45:49.831 959 DEBUG masakari.api.openstack.wsgi [req-e9a58522-858d-4025-9c43-f9fee744a0db nova - - - -] Returning 400 to user: Compute service with name XXXXX could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
On 11 Jun 2021, at 14:56, Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
On Fri, Jun 11, 2021 at 2:47 PM at <bkslash@poczta.onet.pl> wrote:
Hi, thx for the answer.
openstack compute service list --service nova-compute --host $HOSTNAME so in
openstack segment host create
I should use name which is displayed in "Host" column, right? So that's what I do :(
Yes.
openstack compute service list --service nova-compute
ID Binary Host Zone Status State 20 nova-compute XXXXX nova enabled up
openstack segment host create XXXXX COMPUTE SSH 00dd5bxxxxxx
and still "Compute service with name XXXXX could not be found".....
How masakari discovers hosts?
I wrote this already: openstack compute service list --service nova-compute --host $HOSTNAME did you try including the same hostname in this command? If it works and Masakari does not, I would make sure you set up Masakari to speak to the right Nova API.
Finally, if all else fails, please paste (e.g. https://paste.ubuntu.com/ ) masakari api logs for those rejected host creations. Though do that with debug=True in the config [DEFAULT] section.
-yoctozepto
The line
2021-06-11 14:45:49.829 959 INFO masakari.api.openstack.wsgi [req-e9a58522-858d-4025-9c43-f9fee744a0db nova - - - -] HTTP exception thrown: Compute service with name XXXXX could not be found.
suggests that nova actively disagrees that this compute node actually exists. As for the exercised behaviour: this is tested both in Masakari and Kolla Ansible CIs, and it works. I am afraid the answer to why this fails lies in the format of that hidden XXXXX. At the moment, I can't really think of how that could affect the outcome. Is XXXXX 100% the same between the different logs? If you can't somehow share how XXXXX looks, then you might want to check the Nova API logs (again, debug=True might help a bit) and compare between how the openstack client query works vs how the Masakari query works. Perhaps, there is a glitch at some stage that causes the XXXXX to get garbled. You can also append --debug to the openstack commands to get the client side of the conversation. On Fri, Jun 11, 2021 at 5:10 PM bkslash <bkslash@poczta.onet.pl> wrote:
openstack compute service list --service nova-compute --host $HOSTNAME did you try including the same hostname in this command? yes, and it returns the same as "openstack compute service list" but of course only for host XXXXX
If it works and Masakari does not, I would make sure you set up Masakari to speak to the right Nova API. I'm using kolla-ansible, all masakari configuration was generated based on globals.yaml and inventory file while deployment, so it should work almost "out of the box". Does masakari speak to nova via RabbitMQ? How else can I check which port/IP masakari speaks to? In logs I can only see requests TO masakari API, not where masakari tries to check hypervisor...
Masakari speaks to Nova via Nova API only. If you used Kolla Ansible, then it's set up correctly unless you manually overrode that. By default, Masakari looks up the Nova endpoint from the Keystone catalogue. -yoctozepto
Problem solved. It was caused by unset (by kolla-ansible role) variable os_region_name in masakari.conf. Problem was visible only if number of regions was > 1. Due to wrong value of os_region_name masakari tried to GET /os-services from wrong endpoint (another region endpoint). Solution: set [DEFAULT] os_region_name = PUT_YOUR_REGION_NAME_HERE in /etc/kolla/config/masakari.conf or edit masakari.conf.j2 template file in kolla-ansible role Radosław Piliszek - thank you for help with this issue. Best Regards Adam Tomas
Wiadomość napisana przez Radosław Piliszek <radoslaw.piliszek@gmail.com> w dniu 14.06.2021, o godz. 09:35:
The line
2021-06-11 14:45:49.829 959 INFO masakari.api.openstack.wsgi [req-e9a58522-858d-4025-9c43-f9fee744a0db nova - - - -] HTTP exception thrown: Compute service with name XXXXX could not be found.
suggests that nova actively disagrees that this compute node actually exists. As for the exercised behaviour: this is tested both in Masakari and Kolla Ansible CIs, and it works. I am afraid the answer to why this fails lies in the format of that hidden XXXXX. At the moment, I can't really think of how that could affect the outcome. Is XXXXX 100% the same between the different logs? If you can't somehow share how XXXXX looks, then you might want to check the Nova API logs (again, debug=True might help a bit) and compare between how the openstack client query works vs how the Masakari query works. Perhaps, there is a glitch at some stage that causes the XXXXX to get garbled. You can also append --debug to the openstack commands to get the client side of the conversation.
On Fri, Jun 11, 2021 at 5:10 PM bkslash <bkslash@poczta.onet.pl> wrote:
openstack compute service list --service nova-compute --host $HOSTNAME did you try including the same hostname in this command? yes, and it returns the same as "openstack compute service list" but of course only for host XXXXX
If it works and Masakari does not, I would make sure you set up Masakari to speak to the right Nova API. I'm using kolla-ansible, all masakari configuration was generated based on globals.yaml and inventory file while deployment, so it should work almost "out of the box". Does masakari speak to nova via RabbitMQ? How else can I check which port/IP masakari speaks to? In logs I can only see requests TO masakari API, not where masakari tries to check hypervisor...
Masakari speaks to Nova via Nova API only. If you used Kolla Ansible, then it's set up correctly unless you manually overrode that. By default, Masakari looks up the Nova endpoint from the Keystone catalogue.
-yoctozepto
On Thu, Aug 19, 2021 at 9:24 AM Adam Tomas <bkslash@poczta.onet.pl> wrote:
Problem solved. It was caused by unset (by kolla-ansible role) variable os_region_name in masakari.conf. Problem was visible only if number of regions was > 1. Due to wrong value of os_region_name masakari tried to GET /os-services from wrong endpoint (another region endpoint).
Solution: set [DEFAULT] os_region_name = PUT_YOUR_REGION_NAME_HERE in /etc/kolla/config/masakari.conf
or edit masakari.conf.j2 template file in kolla-ansible role
Radosław Piliszek - thank you for help with this issue.
Thank you for the summary. I will only add that we are tracking this as Kolla Ansible bug in https://bugs.launchpad.net/kolla-ansible/+bug/1939291 Fixed soon. -yoctozepto
Best Regards Adam Tomas
Wiadomość napisana przez Radosław Piliszek <radoslaw.piliszek@gmail.com> w dniu 14.06.2021, o godz. 09:35:
The line
2021-06-11 14:45:49.829 959 INFO masakari.api.openstack.wsgi [req-e9a58522-858d-4025-9c43-f9fee744a0db nova - - - -] HTTP exception thrown: Compute service with name XXXXX could not be found.
suggests that nova actively disagrees that this compute node actually exists. As for the exercised behaviour: this is tested both in Masakari and Kolla Ansible CIs, and it works. I am afraid the answer to why this fails lies in the format of that hidden XXXXX. At the moment, I can't really think of how that could affect the outcome. Is XXXXX 100% the same between the different logs? If you can't somehow share how XXXXX looks, then you might want to check the Nova API logs (again, debug=True might help a bit) and compare between how the openstack client query works vs how the Masakari query works. Perhaps, there is a glitch at some stage that causes the XXXXX to get garbled. You can also append --debug to the openstack commands to get the client side of the conversation.
On Fri, Jun 11, 2021 at 5:10 PM bkslash <bkslash@poczta.onet.pl> wrote:
openstack compute service list --service nova-compute --host $HOSTNAME did you try including the same hostname in this command? yes, and it returns the same as "openstack compute service list" but of course only for host XXXXX
If it works and Masakari does not, I would make sure you set up Masakari to speak to the right Nova API. I'm using kolla-ansible, all masakari configuration was generated based on globals.yaml and inventory file while deployment, so it should work almost "out of the box". Does masakari speak to nova via RabbitMQ? How else can I check which port/IP masakari speaks to? In logs I can only see requests TO masakari API, not where masakari tries to check hypervisor...
Masakari speaks to Nova via Nova API only. If you used Kolla Ansible, then it's set up correctly unless you manually overrode that. By default, Masakari looks up the Nova endpoint from the Keystone catalogue.
-yoctozepto
participants (4)
-
Adam Tomas
-
at
-
bkslash
-
Radosław Piliszek