[ironic][nova][neutron][cloud-init] Infiniband Support in OpenStack
Hi,
I am trying to deploy a node with infiniband in Ironic without any success.
The node has two interfaces, eth0 and ib0. The deployment is successful, node becomes active but is not reachable. I debugged and checked that the issue is with cloud-init. The cloud-init fails to configure the network interfaces on the node complaining that the MAC address of infiniband port(ib0) is not known to the node. Ironic provides a fake MAC address for infiniband ports and cloud-init is supposed to generate the actual MAC address of infiband ports[1]. But it fails[2] before reaching there. I have posted the issue in cloud-init[3] as well.
Can someone please help me with this issue? How do we specify "TYPE=InfiniBand" from OpenStack? Currently the type sent is "phy" only.
[1] https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/helper... [2] https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/helper... [3] https://bugs.launchpad.net/cloud-init/+bug/1857031
Regards, Madhuri
On Fri, Jan 17, 2020, at 7:31 AM, Kumari, Madhuri wrote:
Hi,
I am trying to deploy a node with infiniband in Ironic without any success.
The node has two interfaces, eth0 and ib0. The deployment is successful, node becomes active but is not reachable. I debugged and checked that the issue is with cloud-init. The cloud-init fails to configure the network interfaces on the node complaining that the MAC address of infiniband port(ib0) is not known to the node. Ironic provides a fake MAC address for infiniband ports and cloud-init is supposed to generate the actual MAC address of infiband ports[1]. But it fails[2] before reaching there.
Reading the cloud-init code [4][5] it appears that the ethernet format MAC should match bytes 13-15 + 18-20 of the infiniband address. Is the problem here that the fake MAC supplied is unrelated to the actual infiniband address? If so I think you'll either need cloud-init to ignore unknown interfaces (as proposed in the cloud-init bug), or have Ironic supply the mac address as bytes 13-15 + 18-20 of the actual infiniband address.
I have posted the issue in cloud-init[3] as well.
Can someone please help me with this issue? How do we specify “TYPE=InfiniBand” from OpenStack? Currently the type sent is “phy” only.
[1] https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/helper...
[2] https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/helper...
[4] https://github.com/canonical/cloud-init/blob/9bfb2ba7268e2c3c932023fc3d3020c... [5] https://github.com/canonical/cloud-init/blob/9bfb2ba7268e2c3c932023fc3d3020c...
Hi Clark,
Thank you for your response. I think the infiniband MAC address should be mac[36:-14] + mac[51:] as suggested here[1]. I have specified the right MAC address as per this but it still fails. Please see the following output:
ib0 interface details: 4: ib0: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN group default qlen 256 link/infiniband 80:00:00:02:fe:80:00:00:00:00:00:00:00:11:75:01:01:67:0f:b9 brd 00:ff:ff:ff:ff:12:40:1b:80:00:00:00:00:00:00:00:ff:ff:ff:ff
Ironic port details: +-----------------------+----------------------------------------------------------------+ | Field | Value | +-----------------------+----------------------------------------------------------------+ | address | 00:11:75:67:0f:b9 | | created_at | 2020-01-16T08:24:15+00:00 | | extra | {'client-id': '0xfe800000000000000011750101670fb9'} | | internal_info | {'tenant_vif_port_id': 'c71b2b31-4231-423c-b512-962623220ddf'} | | is_smartnic | False | | local_link_connection | {} | | node_uuid | 05cce921-931f-4755-ad87-fc41d79a8988 | | physical_network | None | | portgroup_uuid | None | | pxe_enabled | False | | updated_at | 2020-01-16T08:59:47+00:00 | | uuid | 9921139a-63cc-4456-8e85-f7673c5c2b3b | +-----------------------+----------------------------------------------------------------+
[1] https://github.com/canonical/cloud-init/blob/9bfb2ba7268e2c3c932023fc3d3020c...https://github.com/canonical/cloud-init/blob/9bfb2ba7268e2c3c932023fc3d3020cdc6d6cc18/cloudinit/net/__init__.py
-----Original Message----- From: Clark Boylan cboylan@sapwetik.org Sent: Saturday, January 18, 2020 3:41 AM To: openstack-discuss@lists.openstack.org Subject: Re: [ironic][nova][neutron][cloud-init] Infiniband Support in OpenStack
On Fri, Jan 17, 2020, at 7:31 AM, Kumari, Madhuri wrote:
Hi,
I am trying to deploy a node with infiniband in Ironic without any success.
The node has two interfaces, eth0 and ib0. The deployment is successful, node becomes active but is not reachable. I debugged and checked that the issue is with cloud-init. The cloud-init fails to configure the network interfaces on the node complaining that the MAC address of infiniband port(ib0) is not known to the node. Ironic provides a fake MAC address for infiniband ports and cloud-init is supposed to generate the actual MAC address of infiband ports[1]. But it fails[2] before reaching there.
Reading the cloud-init code [4][5] it appears that the ethernet format MAC should match bytes 13-15 + 18-20 of the infiniband address. Is the problem here that the fake MAC supplied is unrelated to the actual infiniband address? If so I think you'll either need cloud-init to ignore unknown interfaces (as proposed in the cloud-init bug), or have Ironic supply the mac address as bytes 13-15 + 18-20 of the actual infiniband address.
I have posted the issue in cloud-init[3] as well.
Can someone please help me with this issue? How do we specify “TYPE=InfiniBand” from OpenStack? Currently the type sent is “phy” only.
[1] https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/ helpers/openstack.py#L686
[2] https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/ helpers/openstack.py#L677
[4] https://github.com/canonical/cloud-https://github.com/canonical/cloud-init/blob/9bfb2ba7268e2c3c932023fc3d3020cdc6d6cc18/cloudinit/net/__init__.py#L793-L795 init/blob/9bfb2ba7268e2c3c932023fc3d3020cdc6d6cc18/cloudinit/net/__init __.py#L793-L795 [5] https://github.com/canonical/cloud-https://github.com/canonical/cloud-init/blob/9bfb2ba7268e2c3c932023fc3d3020cdc6d6cc18/cloudinit/net/__init__.py#L844-L846 init/blob/9bfb2ba7268e2c3c932023fc3d3020cdc6d6cc18/cloudinit/net/__init __.py#L844-L846
On Fri, 17 Jan 2020 at 15:34, Kumari, Madhuri madhuri.kumari@intel.com wrote:
Hi,
I am trying to deploy a node with infiniband in Ironic without any success.
The node has two interfaces, eth0 and ib0. The deployment is successful, node becomes active but is not reachable. I debugged and checked that the issue is with cloud-init. The cloud-init fails to configure the network interfaces on the node complaining that the MAC address of infiniband port(ib0) is not known to the node. Ironic provides a fake MAC address for infiniband ports and cloud-init is supposed to generate the actual MAC address of infiband ports[1]. But it fails[2] before reaching there.
I have posted the issue in cloud-init[3] as well.
Can someone please help me with this issue? How do we specify “TYPE=InfiniBand” from OpenStack? Currently the type sent is “phy” only.
[1] https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/helper...
[2] https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/helper...
Hi Madhuri,
Please see my blog post: https://www.stackhpc.com/bare-metal-infiniband.html. One major question to ask is whether you want shared IB network or multi-tenant isolation. The latter is significantly more challenging. It's probably best if you read that article and raise any further questions here or IRC. I'll be out of the office until Wednesday.
Mark
Regards,
Madhuri
Hi Mark,
Thanks for your response. I read the blog and my requirement is shared IB network as of now. We have developed a Neutron ML2 driver that manages the IB subnet[1].
Regardless of that, I think my current issue is because the cloud-init is not able to recognize the port as IB port. How did it work for you? You can see my ironic port details.
[1] https://opendev.org/x/networking-omnipath/
-----Original Message-----
From: Mark Goddard mark@stackhpc.com
Sent: Monday, January 20, 2020 2:10 PM
To: Kumari, Madhuri madhuri.kumari@intel.com
Cc: openstack-discuss@lists.openstack.org
Subject: Re: [ironic][nova][neutron][cloud-init] Infiniband Support in
OpenStack
On Fri, 17 Jan 2020 at 15:34, Kumari, Madhuri <madhuri.kumari@intel.commailto:madhuri.kumari@intel.com>
wrote:
Hi,
I am trying to deploy a node with infiniband in Ironic without any success.
The node has two interfaces, eth0 and ib0. The deployment is successful,
node becomes active but is not reachable. I debugged and checked that the
issue is with cloud-init. The cloud-init fails to configure the network interfaces
on the node complaining that the MAC address of infiniband port(ib0) is not
known to the node. Ironic provides a fake MAC address for infiniband ports
and cloud-init is supposed to generate the actual MAC address of infiband
ports[1]. But it fails[2] before reaching there.
I have posted the issue in cloud-init[3] as well.
Can someone please help me with this issue? How do we specify
“TYPE=InfiniBand” from OpenStack? Currently the type sent is “phy” only.
[1]
https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/
helpers/openstack.py#L686
[2]
https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/
helpers/openstack.py#L677
Hi Madhuri,
Please see my blog post:
https://www.stackhpc.com/bare-metal-infiniband.html. One major question
to ask is whether you want shared IB network or multi-tenant isolation. The
latter is significantly more challenging. It's probably best if you read that
article and raise any further questions here or IRC. I'll be out of the office
until Wednesday.
Mark
Regards,
Madhuri
On Wed, 22 Jan 2020 at 08:37, Kumari, Madhuri madhuri.kumari@intel.com wrote:
Hi Mark,
Thanks for your response. I read the blog and my requirement is shared IB network as of now. We have developed a Neutron ML2 driver that manages the IB subnet[1].
Regardless of that, I think my current issue is because the cloud-init is not able to recognize the port as IB port. How did it work for you? You can see my ironic port details.
I see, you didn't mention that you are using Omnipath which is based on IB but has differences.
Here's the relevant code in cloudinit: https://github.com/canonical/cloud-init/blob/9bfb2ba7268e2c3c932023fc3d3020c.... In order for an interface to be picked up, it needs to have a type of 32 and the ethernet format MAC address of the port must match bytes 13-15 and 18-20 of the IB hardware address.
-----Original Message-----
From: Mark Goddard mark@stackhpc.com
Sent: Monday, January 20, 2020 2:10 PM
To: Kumari, Madhuri madhuri.kumari@intel.com
Cc: openstack-discuss@lists.openstack.org
Subject: Re: [ironic][nova][neutron][cloud-init] Infiniband Support in
OpenStack
On Fri, 17 Jan 2020 at 15:34, Kumari, Madhuri madhuri.kumari@intel.com
wrote:
Hi,
I am trying to deploy a node with infiniband in Ironic without any success.
The node has two interfaces, eth0 and ib0. The deployment is successful,
node becomes active but is not reachable. I debugged and checked that the
issue is with cloud-init. The cloud-init fails to configure the network interfaces
on the node complaining that the MAC address of infiniband port(ib0) is not
known to the node. Ironic provides a fake MAC address for infiniband ports
and cloud-init is supposed to generate the actual MAC address of infiband
ports[1]. But it fails[2] before reaching there.
I have posted the issue in cloud-init[3] as well.
Can someone please help me with this issue? How do we specify
“TYPE=InfiniBand” from OpenStack? Currently the type sent is “phy” only.
[1]
https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/
helpers/openstack.py#L686
[2]
https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/
helpers/openstack.py#L677
Hi Madhuri,
Please see my blog post:
https://www.stackhpc.com/bare-metal-infiniband.html. One major question
to ask is whether you want shared IB network or multi-tenant isolation. The
latter is significantly more challenging. It's probably best if you read that
article and raise any further questions here or IRC. I'll be out of the office
until Wednesday.
Mark
Regards,
Madhuri
participants (3)
-
Clark Boylan
-
Kumari, Madhuri
-
Mark Goddard