On 2024-09-23 18:42:43 -0400 (-0400), Michael Knox wrote:
For our clouds, we always use jumbo frames (9000) for the tenant networks. We have one cloud with a lot of Oracle RAC workloads, the large MTU helps a lot, it was a negative to use 1500 during the build out of it. Our switches are 25Gb by default, if that matters. As for our management, tooling networks, we stick with 1500, likewise with our Ceph networks. I know in a past life, where I was doing a lot of VMware, it was always jumbo frames. The layered tenant networks tend to benefit from a larger MTU. [...]
My point was, the guest interfaces can't be 1500 MTU when their Ethernet frames are tunneled over another protocol, unless the underlying network on which they reside supports frame sizes sufficiently *larger* than that in order to accommodate the additional protocol headers such tunneling implies. When I was last doing these things, it was typically handled by setting the physical switching hardware for jumbo frames, and that way the outermost frames could be plenty large enough to handle the additional overhead of tunneling 1500 MTU Ethernet inside the inner protocol(s). 9000-byte frame sizes are pretty huge for this particular purpose, but back then a lot of switch gear either did "traditional" frame sizes or "jumbo" (with nothing in between), and so later on we tended to call any larger nontraditional frame size settings "jumbo frame Ethernet" even in cases where they weren't necessarily a full 9000 bytes. -- Jeremy Stanley