Can you see the dhcp packets in the network namespace dnsmasq is running in?
ip netns list ip netns exec <namespace> ip addr ip netns exec <namespace> tcpdump -i <interface from namespace> <add your filters and remaining tcpdump parameters here>
On Thu, Mar 31, 2022 at 2:23 AM 韩光宇 hanguangyu2@gmail.com wrote:
Hi all,
I have a OpenStack Victoria All-in-One environment, and I can deploy os to bare metal by Ironic. I use `openstack server create --flavor my-baremetal-flavor --nic net-id=$PROVISION_UUID --image $whiole-disk-image BM-server-01` to get a server successfully.
But if I delete server by "openstack server delete BM-server-01", baremetal node will be clean. When baremetal node performs the clean process, it enters the PXE phase, but its DHCP request cannot be replied.
The log of ironic-conductor: 2022-03-31 11:40:42.572 63976 INFO ironic.conductor.task_manager [req-9e92ea87-d48c-42ed-83fd-2f7f14a20984 8763be6de0434240803f305c895fced5 a1758405f9c24ae191543248ad65a447 - default default] Node 4b243894-d651-4700-9a69-28ca3c60446f moved to provision state "clean wait" from state "cleaning"; target provision state is "available" 2022-03-31 12:11:06.694 63976 ERROR ironic.conductor.task_manager [req-3ff5cebf-0895-4e85-bf75-4ac1d20b30f4 - - - - -] Node 4b243894-d651-4700-9a69-28ca3c60446f moved to provision state "clean failed" from state "clean wait"; target provision state is "available" 2022-03-31 12:11:07.676 63976 INFO ironic.drivers.modules.network.flat [-] Removing ports from cleaning network for node 4b243894-d651-4700-9a69-28ca3c60446f 2022-03-31 12:11:08.218 63976 INFO ironic.common.neutron [-] Successfully removed node 4b243894-d651-4700-9a69-28ca3c60446f neutron ports. 2022-03-31 12:11:08.222 63976 ERROR ironic.conductor.utils [-] Timeout reached while cleaning the node. Please check if the ramdisk responsible for the cleaning is running on the node. Failed on step {}.
The log of tcpdump in openstack: # tcpdump -n -S -i br-ex port 67 or port 68 or port 69 11:44:43.358657 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 18:9b:a5:81:46:b7, length 300 11:44:46.550963 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 18:9b:a5:81:46:b7, length 300 11:44:51.558266 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 18:9b:a5:81:46:b7, length 300 11:44:57.374583 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 18:9b:a5:81:46:b7, length 300 11:44:57.375096 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 18:9b:a5:81:46:b7, length 300 11:44:57.691646 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 18:9b:a5:81:46:b7, length 300 11:44:57.807714 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 18:9b:a5:81:46:b7, length 300 11:44:57.907428 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 18:9b:a5:81:46:b7, length 300 11:44:58.027351 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 18:9b:a5:81:46:b7, length 300
The log of /var/log/neutron/dhcp-agent.log: 2022-03-31 11:40:18.373 47489 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=, binding:profile=, binding:vif_details=, binding:vif_type=unbound, binding:vnic_type=baremetal, created_at=2022-03-31T03:40:18Z, description=, device_id=4b243894-d651-4700-9a69-28ca3c60446f, device_owner=, extra_dhcp_opts=[], fixed_ips=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d29aeeb8>], id=20e8d806-f976-4415-85f5-ed2df55978dc, mac_address=fa:16:3e:f1:b3:aa, name=, network=admin_state_up=True, availability_zone_hints=[], availability_zones=['nova'], created_at=2022-03-28T08:39:24Z, description=, id=faa2b805-7898-4fb2-ad51-49251da6e35e, ipv4_address_scope=None, ipv6_address_scope=None, is_default=False, mtu=1500, name=provider1, port_security_enabled=True, project_id=a1758405f9c24ae191543248ad65a447, provider:network_type=flat, provider:physical_network=provider, provider:segmentation_id=None, revision_number=2, router:external=True, shared=True, status=ACTIVE, subnets=['2d5d4c2c-8c8c-44ef-92b4-a124ed52030f'], tags=[], tenant_id=a1758405f9c24ae191543248ad65a447, updated_at=2022-03-28T08:39:29Z, vlan_transparent=None, network_id=faa2b805-7898-4fb2-ad51-49251da6e35e, port_security_enabled=True, project_id=a1758405f9c24ae191543248ad65a447, revision_number=1, security_groups=['f574b5d0-8308-40f4-a8e5-81126b920e8f'], status=DOWN, tags=[], tenant_id=a1758405f9c24ae191543248ad65a447, updated_at=2022-03-31T03:40:18Z on network _ns_name=qdhcp-faa2b805-7898-4fb2-ad51-49251da6e35e, admin_state_up=True, availability_zone_hints=[], availability_zones=['nova'], created_at=2022-03-28T08:39:24Z, description=, id=faa2b805-7898-4fb2-ad51-49251da6e35e, ipv4_address_scope=None, ipv6_address_scope=None, is_default=False, mtu=1500, name=provider1, non_local_subnets=[], port_security_enabled=True, ports=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d3a16a98>], project_id=a1758405f9c24ae191543248ad65a447, provider:network_type=flat, provider:physical_network=provider, provider:segmentation_id=None, revision_number=2, router:external=True, shared=True, status=ACTIVE, subnets=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d3a169d8>], tags=[], tenant_id=a1758405f9c24ae191543248ad65a447, updated_at=2022-03-28T08:39:29Z, vlan_transparent=None 2022-03-31 11:40:18.516 47489 INFO neutron.agent.dhcp.agent [req-9b1713fa-1082-4ccf-8c8b-e953beaf5e5d - - - - -] DHCP configuration for ports {'20e8d806-f976-4415-85f5-ed2df55978dc'} is completed 2022-03-31 11:40:19.517 47489 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=4b243894-d651-4700-9a69-28ca3c60446f, binding:profile=local_link_information=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d29aeb28>], binding:vif_details=, binding:vif_type=unbound, binding:vnic_type=baremetal, created_at=2022-03-31T03:40:18Z, description=, device_id=4b243894-d651-4700-9a69-28ca3c60446f, device_owner=baremetal:none, extra_dhcp_opts=[], fixed_ips=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d29aefa8>], id=20e8d806-f976-4415-85f5-ed2df55978dc, mac_address=18:9b:a5:81:46:b7, name=, network_id=faa2b805-7898-4fb2-ad51-49251da6e35e, port_security_enabled=True, project_id=a1758405f9c24ae191543248ad65a447, revision_number=2, security_groups=['f574b5d0-8308-40f4-a8e5-81126b920e8f'], status=DOWN, tags=[], tenant_id=a1758405f9c24ae191543248ad65a447, updated_at=2022-03-31T03:40:19Z on network _ns_name=qdhcp-faa2b805-7898-4fb2-ad51-49251da6e35e, admin_state_up=True, availability_zone_hints=[], availability_zones=['nova'], created_at=2022-03-28T08:39:24Z, description=, id=faa2b805-7898-4fb2-ad51-49251da6e35e, ipv4_address_scope=None, ipv6_address_scope=None, is_default=False, mtu=1500, name=provider1, non_local_subnets=[], port_security_enabled=True, ports=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d3a16a98>, <neutron.agent.linux.dhcp.DictModel object at 0x7f56d29aeaf8>], project_id=a1758405f9c24ae191543248ad65a447, provider:network_type=flat, provider:physical_network=provider, provider:segmentation_id=None, revision_number=2, router:external=True, shared=True, status=ACTIVE, subnets=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d3a169d8>], tags=[], tenant_id=a1758405f9c24ae191543248ad65a447, updated_at=2022-03-28T08:39:29Z, vlan_transparent=None 2022-03-31 11:40:19.649 47489 INFO neutron.agent.dhcp.agent [req-7b9253ec-0a60-4309-ac66-8e3ea167cbba - - - - -] DHCP configuration for ports {'20e8d806-f976-4415-85f5-ed2df55978dc'} is completed 2022-03-31 11:40:19.678 47489 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=4b243894-d651-4700-9a69-28ca3c60446f, binding:profile=local_link_information=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d29c88e8>], binding:vif_details=, binding:vif_type=binding_failed, binding:vnic_type=baremetal, created_at=2022-03-31T03:40:18Z, description=, device_id=4b243894-d651-4700-9a69-28ca3c60446f, device_owner=baremetal:none, extra_dhcp_opts=[], fixed_ips=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d29c8558>], id=20e8d806-f976-4415-85f5-ed2df55978dc, mac_address=18:9b:a5:81:46:b7, name=, network_id=faa2b805-7898-4fb2-ad51-49251da6e35e, port_security_enabled=True, project_id=a1758405f9c24ae191543248ad65a447, revision_number=3, security_groups=['f574b5d0-8308-40f4-a8e5-81126b920e8f'], status=DOWN, tags=[], tenant_id=a1758405f9c24ae191543248ad65a447, updated_at=2022-03-31T03:40:19Z on network _ns_name=qdhcp-faa2b805-7898-4fb2-ad51-49251da6e35e, admin_state_up=True, availability_zone_hints=[], availability_zones=['nova'], created_at=2022-03-28T08:39:24Z, description=, id=faa2b805-7898-4fb2-ad51-49251da6e35e, ipv4_address_scope=None, ipv6_address_scope=None, is_default=False, mtu=1500, name=provider1, non_local_subnets=[], port_security_enabled=True, ports=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d3a16a98>, <neutron.agent.linux.dhcp.DictModel object at 0x7f56d28b53d8>], project_id=a1758405f9c24ae191543248ad65a447, provider:network_type=flat, provider:physical_network=provider, provider:segmentation_id=None, revision_number=2, router:external=True, shared=True, status=ACTIVE, subnets=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d3a169d8>], tags=[], tenant_id=a1758405f9c24ae191543248ad65a447, updated_at=2022-03-28T08:39:29Z, vlan_transparent=None 2022-03-31 11:40:19.776 47489 INFO neutron.agent.dhcp.agent [req-99eb793a-230c-490a-b126-ee9b595ed1bf - - - - -] DHCP configuration for ports {'20e8d806-f976-4415-85f5-ed2df55978dc'} is completed 2022-03-31 11:40:21.082 47489 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=4b243894-d651-4700-9a69-28ca3c60446f, binding:profile=local_link_information=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d288ea68>], binding:vif_details=, binding:vif_type=binding_failed, binding:vnic_type=baremetal, created_at=2022-03-31T03:40:18Z, description=, device_id=4b243894-d651-4700-9a69-28ca3c60446f, device_owner=baremetal:none, extra_dhcp_opts=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d288e9a8>, <neutron.agent.linux.dhcp.DictModel object at 0x7f56d288ed38>, <neutron.agent.linux.dhcp.DictModel object at 0x7f56d288e4f8>, <neutron.agent.linux.dhcp.DictModel object at 0x7f56d288e528>, <neutron.agent.linux.dhcp.DictModel object at 0x7f56d288e948>], fixed_ips=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d288e0a8>], id=20e8d806-f976-4415-85f5-ed2df55978dc, mac_address=18:9b:a5:81:46:b7, name=, network_id=faa2b805-7898-4fb2-ad51-49251da6e35e, port_security_enabled=True, project_id=a1758405f9c24ae191543248ad65a447, revision_number=4, security_groups=['f574b5d0-8308-40f4-a8e5-81126b920e8f'], status=DOWN, tags=[], tenant_id=a1758405f9c24ae191543248ad65a447, updated_at=2022-03-31T03:40:21Z on network _ns_name=qdhcp-faa2b805-7898-4fb2-ad51-49251da6e35e, admin_state_up=True, availability_zone_hints=[], availability_zones=['nova'], created_at=2022-03-28T08:39:24Z, description=, id=faa2b805-7898-4fb2-ad51-49251da6e35e, ipv4_address_scope=None, ipv6_address_scope=None, is_default=False, mtu=1500, name=provider1, non_local_subnets=[], port_security_enabled=True, ports=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d3a16a98>, <neutron.agent.linux.dhcp.DictModel object at 0x7f56d29c8eb8>], project_id=a1758405f9c24ae191543248ad65a447, provider:network_type=flat, provider:physical_network=provider, provider:segmentation_id=None, revision_number=2, router:external=True, shared=True, status=ACTIVE, subnets=[<neutron.agent.linux.dhcp.DictModel object at 0x7f56d3a169d8>], tags=[], tenant_id=a1758405f9c24ae191543248ad65a447, updated_at=2022-03-28T08:39:29Z, vlan_transparent=None 2022-03-31 11:40:21.207 47489 INFO neutron.agent.dhcp.agent [req-d7cb57c1-4cda-4ceb-99ca-18ca650db61b - - - - -] DHCP configuration for ports {'20e8d806-f976-4415-85f5-ed2df55978dc'} is completed
I would very appreciate any kind of guidance or help.
Thank you, Han Guangyu