Thanks Julia.

In addition to what you mentioned this particular issue seems to have cropped up when we added 100 more baremetal nodes.

I've also narrowed down the issue (TFTP timeouts) when 3-4 baremetal nodes are in "deploy" state and downloading the OS via iSCSI. Each iSCSI transfer takes about 6 Gbps and thus with four transfers we are over our 20Gbps capacity of the leaf-spine links. We are slowly migrating to iPXE so it should help.

That being said is there a document on large scale ironic design architectures?

We are looking into a DC design (primarily for baremetals) for upto 2500 nodes.

thanks,

Fred,

On Wednesday, October 23, 2019, 03:19:41 PM PDT, Julia Kreger <juliaashleykreger@gmail.com> wrote:

Greetings Fred!

Reply in-line.

On Tue, Oct 22, 2019 at 12:47 PM fsbiz@yahoo.com <fsbiz@yahoo.com> wrote:

[trim]

TFTP logs: shows TFTP client timed out (weird). Any pointers here?

Sadly this is one of those things that comes with using TFTP. Issues like this is why the community tends to recommend using ipxe.efi to chainload as you can perform transport over TCP as opposed to UDP where in something might happen mid-transport.

tftpd shows ramdisk_deployed completed. Then, it reports that the client timed out.

Grub does tend to be very abrupt and not wrap up very final actions. I suspect it may just never be sending the ack back and the transfer may be completing. I'm afraid this is one of those things you really need to see on the console what is going on. My guess would be that your deploy_ramdisk lost a packet in transfer or that it was corrupted in transport. It would be interesting to know if the network card stack is performing checksum validation, but for IPv4 it is optional.

[trim]

This has me stumped here. This exact failure seems to be happening 3 to 4 times a week on different nodes.
Any pointers appreciated.

thanks,
Fred.