<div dir="ltr"><div>Hi Mark,</div><div><br></div><div>Earlier I was passing the boot_mode as uefi while creating the baremetal node. </div><div>On Kolla-Ansible Launchpad, I found some issues related to UEFI mode, so I didn't pass the parameter.</div><div><br></div><div>With IPXE and without passing UEFI boot mode parameter, my node started cleaning. It connected with the TFTP server.</div><div><br></div><div>But from the last 2 hours, the state is still in <b>clean_wait</b> only.</div><div><br></div><div>The ramdisk and kernel images I used were the ones mentioned in the link below</div><div><br></div><ul><li><a href="https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.kernel">https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.kernel</a></li><li><a href="https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.initramfs">https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.initramfs</a></li></ul><div><br></div><div>For this I followed the latest kolla ansible document:-</div><div><ul><li><a href="https://docs.openstack.org/kolla-ansible/latest/reference/bare-metal/ironic-guide.html">https://docs.openstack.org/kolla-ansible/latest/reference/bare-metal/ironic-guide.html</a></li></ul></div><div><br></div><div>All I can see in <b>ironic-conductor</b> logs is:</div><div><br></div><div>2021-08-09 13:49:51.159 7 DEBUG ironic.drivers.modules.agent_base [-] Heartbeat from node 8b1ec553-fbc9-4912-bd33-88afc41b8f81 heartbeat /var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/agent_base.py:641<br>2021-08-09 13:49:51.178 7 DEBUG ironic.drivers.modules.agent_client [-] Fetching status of agent commands for node 8b1ec553-fbc9-4912-bd33-88afc41b8f81 get_commands_status /var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/agent_client.py:310<br>2021-08-09 13:49:51.186 7 DEBUG ironic.drivers.modules.agent_client [-] Status of agent commands for node 8b1ec553-fbc9-4912-bd33-88afc41b8f81: get_clean_steps: result "{'clean_steps': {'GenericHardwareManager': [{'step': 'erase_devices', 'priority': 10, 'interface': 'deploy', 'reboot_requested': False, 'abortable': True}, {'step': 'erase_devices_metadata', 'priority': 99, 'interface': 'deploy', 'reboot_requested': False, 'abortable': True}, {'step': 'erase_pstore', 'priority': 0, 'interface': 'deploy', 'reboot_requested': False, 'abortable': True}, {'step': 'delete_configuration', 'priority': 0, 'interface': 'raid', 'reboot_requested': False, 'abortable': True}, {'step': 'create_configuration', 'priority': 0, 'interface': 'raid', 'reboot_requested': False, 'abortable': True}, {'step': 'burnin_cpu', 'priority': 0, 'interface': 'deploy', 'reboot_requested': False, 'abortable': True}, {'step': 'burnin_disk', 'priority': 0, 'interface': 'deploy', 'reboot_requested': False, 'abortable': True}, {'step': 'burnin_memory', 'priority': 0, 'interface': 'deploy', 'reboot_requested': False, 'abortable': True}, {'step': 'burnin_network', 'priority': 0, 'interface': 'deploy', 'reboot_requested': False, 'abortable': True}]}, 'hardware_manager_version': {'MellanoxDeviceHardwareManager': '1', 'generic_hardware_manager': '1.1'}}", error "None"; execute_clean_step: result "{'clean_result': None, 'clean_step': {'step': 'erase_devices_metadata', 'priority': 99, 'interface': 'deploy', 'reboot_requested': False, 'abortable': True, 'requires_ramdisk': True}}", error "None"; execute_clean_step: result "None", error "None" get_commands_status /var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/agent_client.py:342<br>2021-08-09 13:49:51.186 7 DEBUG ironic.drivers.modules.agent_base [-] <b>Clean step still running for node 8b1ec553-fbc9-4912-bd33-88afc41b8f81:</b> None _get_completed_command /var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/agent_base.py:267<br></div><div><br></div><div>It would be a great help if you could suggest some pointers.</div><div><br></div><div>Regards</div><div>Anirudh Gupta</div><div><br></div><div><br><div><br></div><div><br></div><div>I tried </div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 9, 2021 at 1:43 PM Mark Goddard <<a href="mailto:mark@stackhpc.com">mark@stackhpc.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div style="font-family:verdana,sans-serif"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 6 Aug 2021 at 13:49, Anirudh Gupta <<a href="mailto:anyrude10@gmail.com" target="_blank">anyrude10@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Dmitry,</div><div><br></div><div>I tried taking TCPDUMP while the Baremetal Node was booting up and looked for tftp protocols and found there was some "<b>File Not Found" </b>traces for bootx64.efi</div><div><br></div><div><img src="cid:ii_ks0azqbr1" alt="image.png" width="497" height="75"><br></div><div><br></div><div>Then, I found a related post on openstack Discuss which suggested to enable IPXE</div><div><a href="http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010329.html" target="_blank">http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010329.html</a><br></div><div><br></div><div>After re-deploying the setup with IPXE enabled, i found similar traces now for <b>ipxe.efi file</b></div><div><b><br></b></div><div><img src="cid:ii_ks0b27pn2" alt="image.png" width="497" height="83"><br></div><div><br></div><div>Can you please now suggest what possibly could be a miss in configuration and steps to resolve it.</div></div></blockquote><div><br></div><div style="font-family:verdana,sans-serif">Hi Anirudh,</div><div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif">I'd suggest installing a tftp client on your machine and making some requests. The TFTP daemon runs in the ironic_pxe container, and TFTP files are served from /tftpboot in that container.</div><div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif">Mark</div><div style="font-family:verdana,sans-serif"></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>For your reference, I am attaching the complete tcpdump logs of both the Scenarios</div><div><br></div><div>Looking forward to hearing from you.</div><div><br></div><div>Regards</div><div>Anirudh Gupta</div><div><br></div><div><br></div><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 5, 2021 at 4:56 PM Anirudh Gupta <<a href="mailto:anyrude10@gmail.com" target="_blank">anyrude10@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Hi Team,</div><div><br></div><div>On further debugging, I found an error in neutron-server logs</div><div><br></div><div><br></div><div><div style="box-sizing:border-box;font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px">Failed to bind port 476d8175-ffc2-49ba-bb12-0a77c1f07e5f on host f4a43fa5-9c41-488e-a34d-714ae5a9d300 for vnic_type baremetal using segments [{'id': '1a5bbe96-2488-4971-925f-7c9346ba3ef5', 'network_type': 'flat', 'physical_network': 'physnet1', 'segmentation_id': None, 'network_id': '5b6cccec-ad86-4ed9-8d3c-72a31ec3a0d4'}]<br>
2021-08-05 16:33:06.979 23 INFO neutron.plugins.ml2.plugin [req-54d11d51-7319-43ea-b70c-fe39d8aafe8a 21d6a238438e4294912746bcdc895e31 3eca725754e1405eb178cc39bd0da3aa - default default] Attempt 9 to bind port 476d8175-ffc2-49ba-bb12-0a77c1f07e5f</div></div><div style="box-sizing:border-box;font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px"><br></div><div style="box-sizing:border-box;font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px">where 476d8175-ffc2-49ba-bb12-0a77c1f07e5f is the uuid of Baremetal Node</div><div style="box-sizing:border-box;font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px"><br></div><div style="box-sizing:border-box;font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px">However the port is created in openstack, but its state is down</div><div style="box-sizing:border-box;font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px"><br></div><div style="box-sizing:border-box;font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px">[ansible@localhost ~]$ openstack port list<br>+--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+<br>| ID | Name | MAC Address | Fixed IP Addresses | Status |<br>+--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+<br>| 07d6b83d-d83c-498f-8ba8-b4f21bef7249 | | fa:16:3e:38:05:9d | ip_address='10.0.1.200', subnet_id='7b72c158-2146-4bd6-893b-bd76b4a3e869' | ACTIVE |<br>| 476d8175-ffc2-49ba-bb12-0a77c1f07e5f | | <b>98:f2:b3:3f:72:d8</b> | ip_address='10.0.1.202', subnet_id='7b72c158-2146-4bd6-893b-bd76b4a3e869' | <b>DOWN </b> |<br>+--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+<br></div><div dir="ltr"><br></div><div dir="ltr"><b style="font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px">98:f2:b3:3f:72:d8 </b><span style="font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px">is the mac address of my Baremetal Node on which PXE is enabled.</span><br></div><div dir="ltr"><br></div><div>Can someone please help in resolving this issue.</div><div><br></div><div><b>Issue:</b></div><div><b>Node goes in clean_failed from clean_wait.</b> </div><div><br></div><div>Regards</div><div>Anirudh Gupta</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 3, 2021 at 8:32 PM Anirudh Gupta <<a href="mailto:anyrude10@gmail.com" target="_blank">anyrude10@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Dmitry,</div><div><br></div><div>I might be wrong, but as per my understanding if there would be an issue in dnsmasq, then IP 20.20.20.10 would not have been assigned to the machine.</div><div><br></div><div>TCPDUMP logs are as below:</div><div><br></div><div>20:16:58.938089 IP controller.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 312<br>20:17:02.765291 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 98:f2:b3:3f:72:e5 (oui Unknown), length 359<br>20:17:02.766303 IP controller.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 312<br>20:17:26.944378 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 98:f2:b3:3f:72:e5 (oui Unknown), length 347<br>20:17:26.944756 IP controller.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 312<br>20:17:30.763627 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 98:f2:b3:3f:72:e5 (oui Unknown), length 359<br>20:17:30.764620 IP controller.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 312<br>20:17:54.938791 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 98:f2:b3:3f:72:e5 (oui Unknown), length 347<br></div><div><br></div><div>Also the neutron dnsmasq logs and ironic inspector logs are attached in the mail.</div><div><br></div><div>Regards</div><div>Anirudh Gupta</div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 3, 2021 at 7:29 PM Dmitry Tantsur <<a href="mailto:dtantsur@redhat.com" target="_blank">dtantsur@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>You need to check the dnsmasq logs (there are two dnsmasqs: from neutron and from ironic-inspector). tcpdump may also help to determine where the packages are lost.</div><div><br></div><div>Dmitry<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 30, 2021 at 10:29 PM Anirudh Gupta <<a href="mailto:anyrude10@gmail.com" target="_blank">anyrude10@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div>Hi Dmitry</div><div dir="auto"><br></div><div dir="auto">Thanks for your time.</div><div dir="auto"><br></div><div dir="auto">My system is getting IP 20.20.20.10 which is in the range defined in ironic_dnsmasq_dhcp_range field under globals.yml file.</div><div dir="auto"><br></div><div dir="auto">ironic_dnsmasq_dhcp_range: "20.20.20.10,20.20.20.100"</div><div dir="auto"><br></div><div dir="auto">And in the cleaning network (public1), the range defined is 20.20.20.150-20.20.20.200</div><div dir="auto"><br></div><div dir="auto">As per my understanding, these 2 ranges should be mutually exclusive.</div><div dir="auto"><br></div><div dir="auto">Please suggest if my understanding is not correct.</div><div dir="auto"><br></div><div dir="auto">Any suggestions what should I do to resolve this issue?</div><div dir="auto"><br></div><div dir="auto">Regards</div><div dir="auto">Anirudh Gupta</div><div dir="auto"><br></div><div dir="auto"><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Sat, 31 Jul, 2021, 12:06 am Dmitry Tantsur, <<a href="mailto:dtantsur@redhat.com" target="_blank">dtantsur@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 29, 2021 at 6:05 PM Anirudh Gupta <<a href="mailto:anyrude10@gmail.com" rel="noreferrer" target="_blank">anyrude10@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi Team,<br></div><div dir="ltr"><br></div><div>In to the email below, I have some updated information:-</div><div><br></div><div>Earlier the allocation range mentioned in "<b>ironic_dnsmasq_dhcp_range</b>" in globals.yml had an overlapping range with the cleaning network, due to which there was some issue in receiving the DHCP request </div><div><br></div><div>After creating a cleaning network with a separate allocation range, I am successfully getting IP allocated to my Baremetal Node</div><div><ul><li>openstack subnet create subnet1 --network public1 --subnet-range <a href="http://20.20.20.0/24" rel="noreferrer" target="_blank">20.20.20.0/24</a> --allocation-pool start=20.20.20.150,end=20.20.20.200 --ip-version=4 --gateway=20.20.20.1 --dhcp</li></ul></div><div><br></div><div><img src="cid:ii_kroym2be0" alt="image.png" width="541" height="213"><br></div><div><br></div><div>After getting the IP, there is no further action on the node. From "<b>clean_wait</b>", it goes into "<b>clean_failed</b>" state after around half an hour.</div></div></blockquote><div><br></div><div>The IP address is not from the cleaning range, it may come from inspection. You probably need to investigate your network topology, maybe use tcpdump.</div><div><br></div><div>Unfortunately, I'm not fluent in Kolla to say if it can be a bug or not.</div><div><br></div><div>Dmitry<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>On verifying the logs, I could see the below error messages</div><div><br></div><div><ul><li>In <b>/var/log/kolla/ironic/ironic-conductor.log</b>, we observed the following error:</li></ul></div><blockquote style="margin:0px 0px 0px 40px;border:medium none;padding:0px"><div>ERROR ironic.conductor.utils [-] Cleaning for node 3a56748e-a8ca-4dec-a332-ace18e6d494e failed. <b>Timeout reached while cleaning the node. Please check if the ramdisk responsible for the cleaning is running on the node. Failed on step {}.</b></div></blockquote><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:medium none;padding:0px"><div>Note : For Cleaning the node, we have used the below images</div></blockquote><br><blockquote style="margin:0px 0px 0px 40px;border:medium none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:medium none;padding:0px"> <a href="https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.kernel" rel="noreferrer" target="_blank">https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.kernel</a></blockquote></blockquote><blockquote style="margin:0px 0px 0px 40px;border:medium none;padding:0px"><blockquote style="margin:0px 0px 0px 40px;border:medium none;padding:0px"> <a href="https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.initramfs" rel="noreferrer" target="_blank">https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.initramfs</a></blockquote></blockquote><div><ul><li>In /var/log/kolla/nova/nova-compute-ironic.log, we observed the error</li></ul></div><blockquote style="margin:0px 0px 0px 40px;border:medium none;padding:0px"><div>ERROR nova.compute.manager [req-810ffedf-3343-471c-94db-85411984e6cc - - - - -] No compute node record for host controller-ironic: nova.exception_Remote.ComputeHostNotFound_Remote: Compute host controller-ironic could not be found.</div></blockquote><div><br></div><div>Can someone please help in this regard?</div><div><br></div><div>Regards</div><div>Anirudh Gupta</div><div><br></div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 27, 2021 at 12:52 PM Anirudh Gupta <<a href="mailto:anyrude10@gmail.com" rel="noreferrer" target="_blank">anyrude10@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Team,<br><br>We have deployed 2 node kolla ansible <b>12.0.0</b> in order to deploy openstack <b>wallaby</b> release. We have also enabled ironic in order to provision the bare metal nodes.<br><br><div>On each server we have 3 nics <br><ul><li><b>eno1</b> - OAM for external connectivity and endpoint's publicURL</li><li><b>eno2</b> - Mgmt for internal communication between various openstack services.</li><li><b>ens2f0</b> - Data Interface</li></ul></div><div><br>Corresponding to this we have defined the following fields in globals.yml<br><br><ul><li>kolla_base_distro: "centos"</li><li>kolla_install_type: "source"</li><li>openstack_release: "wallaby"</li><li>network_interface: "eno2" # MGMT interface</li><li>kolla_external_vip_interface: "eno1" # OAM Interface</li><li>kolla_internal_vip_address: "192.168.10.3" # MGMT Subnet free ip</li><li>kolla_external_vip_address: "10.0.1.136" # OAM subnet free IP</li><li>neutron_external_interface: "ens2f0" # Data Interface</li><li>enable_neutron_provider_networks: "yes"</li></ul>Note: Only relevant fields are being shown in this query</div><div><br>Also, for ironic following fields have been defined in globals.yml<br><ul><li>enable_ironic: "yes"</li><li>enable_ironic_neutron_agent: "{{ enable_neutron | bool and enable_ironic | bool }}"</li><li>enable_horizon_ironic: "{{ enable_ironic | bool }}"</li><li>ironic_dnsmasq_interface: "<b>ens2f0</b>" # Data interface</li><li>ironic_dnsmasq_dhcp_range: "20.20.20.10,20.20.20.100" </li><li>ironic_dnsmasq_boot_file: "pxelinux.0"</li><li>ironic_cleaning_network: "public1"</li><li>ironic_dnsmasq_default_gateway: "20.20.20.1"</li></ul><br>After successful deployment, a flat provider network with the name public1 is being created in openstack using the below commands:<br><br><ul><li>openstack network create public1 --provider-network-type flat --provider-physical-network physnet1</li><li>openstack subnet create subnet1 --network public1 --subnet-range <a href="http://20.20.20.0/24" rel="noreferrer" target="_blank">20.20.20.0/24</a> --allocation-pool start=20.20.20.10,end=20.20.20.100 --ip-version=4 --gateway=20.20.20.1 --dhcp</li></ul><br>Issue/Queries:<br><br><ul><li>Is the configuration done in globals.yml correct or is there anything else that needs to be done in order to separate control and data plane traffic?</li></ul><ul><li>Also I have set automated_cleaning as "true" in ironic-conductor conatiner settings.But after creating the baremetal node, we run "node manage" command which runs successfully. Running "<b>openstack baremetal node provide <node id>"</b> command powers on the machine, sets the boot mode on Network Boot but no DHCP request for that particular mac is obtained on the controller. Is there anything I am missing that needs to be done in order to make ironic work?</li></ul>Note: I have also verified that the nic is PXE enabled in system configuration setting</div><div><br></div><div>Regards</div><div>Anirudh Gupta<br><br><br></div></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr">Red Hat GmbH, <a href="https://de.redhat.com/" rel="noreferrer" target="_blank">https://de.redhat.com/</a> , Registered seat: Grasbrunn, <br>Commercial register: Amtsgericht Muenchen, HRB 153243,<br>Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill <br></div></div></div>
</blockquote></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr">Red Hat GmbH, <a href="https://de.redhat.com/" target="_blank">https://de.redhat.com/</a> , Registered seat: Grasbrunn, <br>Commercial register: Amtsgericht Muenchen, HRB 153243,<br>Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill <br></div></div>
</blockquote></div></div>
</blockquote></div></div></div>
</blockquote></div></div>
</blockquote></div></div>
</blockquote></div></div></div>