[Nova][Neutron] Help troubleshooting metadata service error 404 (Ussuri)

Jérôme BECOT jerome.becot at deveryware.com
Fri Sep 22 14:10:25 UTC 2023 

It finally turned out that the impacted machines deployed a proxy 
configuration that is loaded in the environment. We added an exception 
for 169.254.169.254 and it now works fine.

Regards

Le 22/09/2023 à 13:41, Jérôme BECOT a écrit :
>
> After spending some time it seems that it is a problem in the 
> instances... but still I don't have a clue: it works when running as 
> root or sudo and doesn't with regular users ...
>
> Le 22/09/2023 à 12:49, Jérôme BECOT a écrit :
>>
>> Hello,
>>
>> I am operating several OpenStack Ussuri clouds and we lately have 
>> random troubles with the metadata service. We run all our networks as 
>> provider networks. The metadata service works fine for months some 
>> hosts can't get their metadata with a 404 error
>>
>> I restarted nova-api, neutron-dhcp-agent and neutron-metadata-agent 
>> on all controllers but we still experience 404 on random nodes:
>>
>> ssh 10.60.51.130
>> $ ip r s
>> ...
>> 169.254.169.254 via 10.60.51.121 dev eth0
>> $ curl -s http://169.254.169.254
>> <html>
>>  <head>
>>   <title>404 Not Found</title>
>>  </head>
>>  <body>
>>   <h1>404 Not Found</h1>
>>   The resource could not be found.<br /><br />
>>  </body>
>> </html>
>>
>> ssh 10.60.51.131
>> $ ip r s
>> ...
>> 169.254.169.254 via 10.60.51.121 dev eth0
>> $ curl -s http://169.254.169.254
>> {"uuid": "ee6e8dea-796b-4a56-bb1e-1c38f4ac9030", "meta": {"Created 
>> By": "terraform"}...}
>>
>> $ grep "10.60.51.131" /var/log/neutron/neutron-metadata-agent.log
>> 2023-09-22 12:41:10.687 21954 INFO eventlet.wsgi.server [-] 
>> 10.60.51.131,<local> "GET /openstack/latest/meta_data.json HTTP/1.1" 
>> status: 200  len: 2729 time: 0.3817778
>> $ grep "10.60.51.130" /var/log/neutron/neutron-metadata-agent.log
>> $ grep "10.60.51.130" /var/log/nova/nova-api.log # on all controllers
>> $ grep "10.60.51.131" /var/log/nova/nova-api.log # on all controllers
>> 2023-09-22 12:41:10.682 57433 INFO nova.metadata.wsgi.server 
>> [req-1d222a75-ccb4-49bc-8b17-2e2e81dc5f9b - - - - -] 
>> 10.60.51.131,1.2.3.4 "GET /openstack/latest/meta_data.json HTTP/1.1" 
>> status: 200 len: 2710 time: 0.2587621
>>
>> Can you help me to troubleshoot this behaviour for some hosts ?
>>
>> Regards
>> -- 
>> *Jérôme B*<https://www.deveryware.com>
-- 
*Jérôme BECOT*
Ingénieur DevOps Infrastructure

Téléphone fixe: 01 82 28 37 06
Mobile : +33 757 173 193
Deveryware - 43 rue Taitbout - 75009 PARIS
https://www.deveryware.com <https://www.deveryware.com>

Deveryware_Logo
<https://www.deveryware.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230922/ecab5052/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: baniere_signature_dw_2022.png
Type: image/png
Size: 471995 bytes
Desc: not available
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230922/ecab5052/attachment-0001.png>

More information about the openstack-discuss mailing list