Slawomir, thanks a lot.

-----邮件原件----- 发件人: Slawomir Kaplonski [mailto:skaplons@redhat.com] 发送时间: 2019年5月16日 18:01 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: [DVR config] Can we use drv_snat agent_mode in every compute node? 重要性: 高

Hi,

According to documentation which You cited even "‘dvr_snat’ - this enables centralized SNAT support in conjunction with DVR”. So yes, dvr_snat will do both, SNAT mode as well as DVR for E-W traffic. We are using it like that in some CI jobs for sure and it works. But I’m not 100% sure that this is “production ready” solution.

On 16 May 2019, at 05:47, Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com wrote:

Hi, folks

I saw somebody discussed distributed SNAT, but finally they didn’t make agreement on how to implement distributed SNAT, my question is can we use dvr_snat agent_mode in compute node? I understand dvr_snat only does snat but doesn’t do east west routing, right? Can we set dvr_snat and dvr in one compute node at the same time? It is equivalent to distributed SNAT if we can set drv_snat in every compute node, isn’t right? I know Opendaylight can do SNAT in compute node in distributed way, but one external router only can run in one compute node.

I also see https://wiki.openstack.org/wiki/Dragonflow is trying to implement distributed SNAT, what are technical road blocks for distributed SNAT in openstack dvr? Do we have any good way to remove these road blocks?

Thank you in advance and look forward to getting your replies and insights.

Also attached official drv configuration guide for your reference.

https://docs.openstack.org/neutron/stein/configuration/l3-agent.html

agent_mode¶ Type string

Default legacy

Valid Values dvr, dvr_snat, legacy, dvr_no_external

The working mode for the agent. Allowed modes are: ‘legacy’ - this preserves the existing behavior where the L3 agent is deployed on a centralized networking node to provide L3 services like DNAT, and SNAT. Use this mode if you do not want to adopt DVR. ‘dvr’ - this mode enables DVR functionality and must be used for an L3 agent that runs on a compute host. ‘dvr_snat’ - this enables centralized SNAT support in conjunction with DVR. This mode must be used for an L3 agent running on a centralized node (or in single-host deployments, e.g. devstack). ‘dvr_no_external’ - this mode enables only East/West DVR routing functionality for a L3 agent that runs on a compute host, the North/South functionality such as DNAT and SNAT will be provided by the centralized network node that is running in ‘dvr_snat’ mode. This mode should be used when there is no external network connectivity on the compute host.

— Slawek Kaplonski Senior software engineer Red Hat

Thanks Brian, your explanation clarified something, but I don't get the answer if we can have multiple compute nodes are configured to dvr_snat, for this case, SNAT IPs are obviously different. Why do we want to use network node if compute node can do everything?

-----邮件原件----- 发件人: Brian Haley [mailto:haleyb.dev@gmail.com] 发送时间: 2019年5月16日 21:46 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: [DVR config] Can we use drv_snat agent_mode in every compute node?

Hi Yi,

I'm a little confused by the question, comments inline.

On 5/15/19 11:47 PM, Yi Yang (杨燚)-云服务集团 wrote:

Hi, folks

I saw somebody discussed distributed SNAT, but finally they didn’t make agreement on how to implement distributed SNAT, my question is can we use dvr_snat agent_mode in compute node? I understand dvr_snat only does snat but doesn’t do east west routing, right? Can we set dvr_snat and dvr in one compute node at the same time? It is equivalent to distributed SNAT if we can set drv_snat in every compute node, isn’t right? I know Opendaylight can do SNAT in compute node in distributed way, but one external router only can run in one compute node.

Distributed SNAT is not available in neutron, there was a spec proposed recently though, https://review.opendev.org/#/c/658414

Regarding the agent_mode setting for L3, only one mode can be set at a time. Typically 'dvr_snat' is used on network nodes and 'dvr' on compute nodes because it leads to less resource usage (i.e. namespaces). The centralized part of the router hosting the default SNAT IP address will only be scheduled to one of the agents in 'dvr_snat' mode. All the DVR modes can do East/West routing when an instance is scheduled to the node, and two can do North/South - 'dvr_snat' using the default SNAT IP, and 'dvr' using a floating IP. 'dvr_no_external' can only do East/West.

Hopefully that clarifies things.

-Brian

I also see https://wiki.openstack.org/wiki/Dragonflow is trying to implement distributed SNAT, what are technical road blocks for distributed SNAT in openstack dvr? Do we have any good way to remove these road blocks?

Thank you in advance and look forward to getting your replies and insights.

Also attached official drv configuration guide for your reference.

https://docs.openstack.org/neutron/stein/configuration/l3-agent.html

|agent_mode|¶ https://docs.openstack.org/neutron/stein/configuration/l3-agent.html# DEFAULT.agent_mode

Type

string

Default

legacy

Valid Values

dvr, dvr_snat, legacy, dvr_no_external

The working mode for the agent. Allowed modes are: ‘legacy’ - this preserves the existing behavior where the L3 agent is deployed on a centralized networking node to provide L3 services like DNAT, and SNAT. Use this mode if you do not want to adopt DVR. ‘dvr’ - this mode enables DVR functionality and must be used for an L3 agent that runs on a compute host. ‘dvr_snat’ - this enables centralized SNAT support in conjunction with DVR. This mode must be used for an L3 agent running on a centralized node (or in single-host deployments, e.g. devstack). ‘dvr_no_external’ - this mode enables only East/West DVR routing functionality for a L3 agent that runs on a compute host, the North/South functionality such as DNAT and SNAT will be provided by the centralized network node that is running in ‘dvr_snat’ mode. This mode should be used when there is no external network connectivity on the compute host.

Brian, thank for your reply. So if I configure 3 compute nodes of many compute node as drv_snat, it doesn't have substantial difference from the case that I configure 3 single network nodes as snat gateway except deployment difference, right? Another question, it doesn't use HA even if we have multiple dvr_snat nodes, right? If enable l3_ha, I think one external router will be scheduled in multiple (2 at least) dvr_snat nodes, for that case, IPs of these HA routers for this one router are same one and are activated by VRRP, right? For l3_ha, two or multiple HA l3 nodes must be in the same L2 network because it uses VRRP (keepalived) to share a VIP, right? For that case, how can we make sure VRRP can work well across leaf switches in a L3 leaf-spine network (servers are connected to leaf switch by L2)?

-----邮件原件----- 发件人: Brian Haley [mailto:haleyb.dev@gmail.com] 发送时间: 2019年5月17日 22:11 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: 答复: [DVR config] Can we use drv_snat agent_mode in every compute node?

On 5/16/19 8:29 PM, Yi Yang (杨燚)-云服务集团 wrote:

Thanks Brian, your explanation clarified something, but I don't get the answer if we can have multiple compute nodes are configured to dvr_snat, for this case, SNAT IPs are obviously different. Why do we want to use network node if compute node can do everything?

Hi Yi,

There will only be one DVR SNAT IP allocated for a router on the external network, and only one router scheduled using it, so having dvr_snat mode on a compute node doesn't mean that North/South router will be local, only the East/West portion might be.

Typically people choose to place these on separate systems since the requirements of the role are different - network node could have fewer cores and a 10G nic for higher bandwidth, compute node could have lots of cores for instances but maybe a 1G nic. There's no reason you can't run dvr_snat everywhere, I would just say it's not common.

-Brian

-----邮件原件----- 发件人: Brian Haley [mailto:haleyb.dev@gmail.com] 发送时间: 2019年5月16日 21:46 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: [DVR config] Can we use drv_snat agent_mode in every compute node?

Hi Yi,

I'm a little confused by the question, comments inline.

On 5/15/19 11:47 PM, Yi Yang (杨燚)-云服务集团 wrote:

...

Hi, folks

I saw somebody discussed distributed SNAT, but finally they didn’t make agreement on how to implement distributed SNAT, my question is can we use dvr_snat agent_mode in compute node? I understand dvr_snat only does snat but doesn’t do east west routing, right? Can we set dvr_snat and dvr in one compute node at the same time? It is equivalent to distributed SNAT if we can set drv_snat in every compute node, isn’t right? I know Opendaylight can do SNAT in compute node in distributed way, but one external router only can run in one compute node.

Distributed SNAT is not available in neutron, there was a spec proposed recently though, https://review.opendev.org/#/c/658414

Regarding the agent_mode setting for L3, only one mode can be set at a time. Typically 'dvr_snat' is used on network nodes and 'dvr' on compute nodes because it leads to less resource usage (i.e. namespaces). The centralized part of the router hosting the default SNAT IP address will only be scheduled to one of the agents in 'dvr_snat' mode. All the DVR modes can do East/West routing when an instance is scheduled to the node, and two can do North/South - 'dvr_snat' using the default SNAT IP, and 'dvr' using a floating IP. 'dvr_no_external' can only do East/West.

Hopefully that clarifies things.

-Brian

...

I also see https://wiki.openstack.org/wiki/Dragonflow is trying to implement distributed SNAT, what are technical road blocks for distributed SNAT in openstack dvr? Do we have any good way to remove these road blocks?

Thank you in advance and look forward to getting your replies and insights.

Also attached official drv configuration guide for your reference.

https://docs.openstack.org/neutron/stein/configuration/l3-agent.html

|agent_mode|¶ https://docs.openstack.org/neutron/stein/configuration/l3-agent.html # DEFAULT.agent_mode

Type

string

Default

legacy

Valid Values

dvr, dvr_snat, legacy, dvr_no_external

The working mode for the agent. Allowed modes are: ‘legacy’ - this preserves the existing behavior where the L3 agent is deployed on a centralized networking node to provide L3 services like DNAT, and SNAT. Use this mode if you do not want to adopt DVR. ‘dvr’ - this mode enables DVR functionality and must be used for an L3 agent that runs on a compute host. ‘dvr_snat’ - this enables centralized SNAT support in conjunction with DVR. This mode must be used for an L3 agent running on a centralized node (or in single-host deployments, e.g. devstack). ‘dvr_no_external’ - this mode enables only East/West DVR routing functionality for a L3 agent that runs on a compute host, the North/South functionality such as DNAT and SNAT will be provided by the centralized network node that is running in ‘dvr_snat’ mode. This mode should be used when there is no external network connectivity on the compute host.

Hi, Slawomir, do you mean VRRP over VXLAN? I mean servers in leaf switch are attached to the leaf switch by VLAN and servers handle VxLAN encap and decap, for such case, how can leaf-spine transport a L2 packet to another server in another leaf switch?

-----邮件原件----- 发件人: Slawomir Kaplonski [mailto:skaplons@redhat.com] 发送时间: 2019年5月20日 15:13 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: haleyb.dev@gmail.com; openstack-discuss@lists.openstack.org 主题: Re: [DVR config] Can we use drv_snat agent_mode in every compute node? 重要性: 高

Hi,

On 20 May 2019, at 02:07, Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com wrote:

Brian, thank for your reply. So if I configure 3 compute nodes of many compute node as drv_snat, it doesn't have substantial difference from the case that I configure 3 single network nodes as snat gateway except deployment difference, right? Another question, it doesn't use HA even if we have multiple dvr_snat nodes, right? If enable l3_ha, I think one external router will be scheduled in multiple (2 at least) dvr_snat nodes, for that case, IPs of these HA routers for this one router are same one and are activated by VRRP, right? For l3_ha, two or multiple HA l3 nodes must be in the same L2 network because it uses VRRP (keepalived) to share a VIP, right? For that case, how can we make sure VRRP can work well across leaf switches in a L3 leaf-spine network (servers are connected to leaf switch by L2)?

That is correct what You are saying. In DVR-HA case, SNAT nodes are working in same way like in “standard” L3HA. So it’s active-backup config and keepalived is deciding which node is active. Neutron creates “HA network” for tenant to use for keepalived. It can be e.g. vxlan network and that way You will have L2 between such nodes (routers).

-----邮件原件----- 发件人: Brian Haley [mailto:haleyb.dev@gmail.com] 发送时间: 2019年5月17日 22:11 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: 答复: [DVR config] Can we use drv_snat agent_mode in every compute node?

On 5/16/19 8:29 PM, Yi Yang (杨燚)-云服务集团 wrote:

...

Thanks Brian, your explanation clarified something, but I don't get the answer if we can have multiple compute nodes are configured to dvr_snat, for this case, SNAT IPs are obviously different. Why do we want to use network node if compute node can do everything?

Hi Yi,

There will only be one DVR SNAT IP allocated for a router on the external network, and only one router scheduled using it, so having dvr_snat mode on a compute node doesn't mean that North/South router will be local, only the East/West portion might be.

Typically people choose to place these on separate systems since the requirements of the role are different - network node could have fewer cores and a 10G nic for higher bandwidth, compute node could have lots of cores for instances but maybe a 1G nic. There's no reason you can't run dvr_snat everywhere, I would just say it's not common.

-Brian

...

-----邮件原件----- 发件人: Brian Haley [mailto:haleyb.dev@gmail.com] 发送时间: 2019年5月16日 21:46 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: [DVR config] Can we use drv_snat agent_mode in every compute node?

Hi Yi,

I'm a little confused by the question, comments inline.

On 5/15/19 11:47 PM, Yi Yang (杨燚)-云服务集团 wrote:

...

Hi, folks

I saw somebody discussed distributed SNAT, but finally they didn’t make agreement on how to implement distributed SNAT, my question is can we use dvr_snat agent_mode in compute node? I understand dvr_snat only does snat but doesn’t do east west routing, right? Can we set dvr_snat and dvr in one compute node at the same time? It is equivalent to distributed SNAT if we can set drv_snat in every compute node, isn’t right? I know Opendaylight can do SNAT in compute node in distributed way, but one external router only can run in one compute node.

Distributed SNAT is not available in neutron, there was a spec proposed recently though, https://review.opendev.org/#/c/658414

Regarding the agent_mode setting for L3, only one mode can be set at a time. Typically 'dvr_snat' is used on network nodes and 'dvr' on compute nodes because it leads to less resource usage (i.e. namespaces). The centralized part of the router hosting the default SNAT IP address will only be scheduled to one of the agents in 'dvr_snat' mode. All the DVR modes can do East/West routing when an instance is scheduled to the node, and two can do North/South - 'dvr_snat' using the default SNAT IP, and 'dvr' using a floating IP. 'dvr_no_external' can only do East/West.

Hopefully that clarifies things.

-Brian

...

I also see https://wiki.openstack.org/wiki/Dragonflow is trying to implement distributed SNAT, what are technical road blocks for distributed SNAT in openstack dvr? Do we have any good way to remove these road blocks?

Thank you in advance and look forward to getting your replies and insights.

Also attached official drv configuration guide for your reference.

https://docs.openstack.org/neutron/stein/configuration/l3-agent.html

|agent_mode|¶ https://docs.openstack.org/neutron/stein/configuration/l3-agent.htm l # DEFAULT.agent_mode

Type

string

Default

legacy

Valid Values

dvr, dvr_snat, legacy, dvr_no_external

The working mode for the agent. Allowed modes are: ‘legacy’ - this preserves the existing behavior where the L3 agent is deployed on a centralized networking node to provide L3 services like DNAT, and SNAT. Use this mode if you do not want to adopt DVR. ‘dvr’ - this mode enables DVR functionality and must be used for an L3 agent that runs on a compute host. ‘dvr_snat’ - this enables centralized SNAT support in conjunction with DVR. This mode must be used for an L3 agent running on a centralized node (or in single-host deployments, e.g. devstack). ‘dvr_no_external’ - this mode enables only East/West DVR routing functionality for a L3 agent that runs on a compute host, the North/South functionality such as DNAT and SNAT will be provided by the centralized network node that is running in ‘dvr_snat’ mode. This mode should be used when there is no external network connectivity on the compute host.

— Slawek Kaplonski Senior software engineer Red Hat

No, please read https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/..., L3HA won't handle VRRP over VXLAN.

1.4. Spine-leaf limitations Some roles, such as the Controller role, use virtual IP addresses and clustering. The mechanism behind this functionality requires layer-2 network connectivity between these nodes. These nodes are all be placed within the same leaf. Similar restrictions apply to Networker nodes. The network service implements highly-available default paths in the network using Virtual Router Redundancy Protocol (VRRP). Since VRRP uses a virtual router IP address, you must connect master and backup nodes to the same L2 network segment. When using tenant or provider networks with VLAN segmentation, you must share the particular VLANs between all Networker and Compute nodes.

-----邮件原件----- 发件人: Slawomir Kaplonski [mailto:skaplons@redhat.com] 发送时间: 2019年5月24日 21:26 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: haleyb.dev@gmail.com; openstack-discuss@lists.openstack.org 主题: Re: [DVR config] Can we use drv_snat agent_mode in every compute node?

Hi,

I’m not expert in spine-leaf topology TBH but I know that for L3HA neutron creates for each tenant, “tenant network” which usually is vxlan or gre tunnels network. And this works like any other vxlan network created in neutron. So tunnels are established between nodes using L3 and it transports “tenant L2” inside vxlan packets, right? In the same way works this network created for L3 HA needs. And it transports VRRP packets inside this tunnel network (which often is vxlan network).

On 20 May 2019, at 09:33, Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com wrote:

Hi, Slawomir, do you mean VRRP over VXLAN? I mean servers in leaf switch are attached to the leaf switch by VLAN and servers handle VxLAN encap and decap, for such case, how can leaf-spine transport a L2 packet to another server in another leaf switch?

-----邮件原件----- 发件人: Slawomir Kaplonski [mailto:skaplons@redhat.com] 发送时间: 2019年5月20日 15:13 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: haleyb.dev@gmail.com; openstack-discuss@lists.openstack.org 主题: Re: [DVR config] Can we use drv_snat agent_mode in every compute node? 重要性: 高

Hi,

...

On 20 May 2019, at 02:07, Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com wrote:

Brian, thank for your reply. So if I configure 3 compute nodes of many compute node as drv_snat, it doesn't have substantial difference from the case that I configure 3 single network nodes as snat gateway except deployment difference, right? Another question, it doesn't use HA even if we have multiple dvr_snat nodes, right? If enable l3_ha, I think one external router will be scheduled in multiple (2 at least) dvr_snat nodes, for that case, IPs of these HA routers for this one router are same one and are activated by VRRP, right? For l3_ha, two or multiple HA l3 nodes must be in the same L2 network because it uses VRRP (keepalived) to share a VIP, right? For that case, how can we make sure VRRP can work well across leaf switches in a L3 leaf-spine network (servers are connected to leaf switch by L2)?

That is correct what You are saying. In DVR-HA case, SNAT nodes are working in same way like in “standard” L3HA. So it’s active-backup config and keepalived is deciding which node is active. Neutron creates “HA network” for tenant to use for keepalived. It can be e.g. vxlan network and that way You will have L2 between such nodes (routers).

...

-----邮件原件----- 发件人: Brian Haley [mailto:haleyb.dev@gmail.com] 发送时间: 2019年5月17日 22:11 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: 答复: [DVR config] Can we use drv_snat agent_mode in every compute node?

On 5/16/19 8:29 PM, Yi Yang (杨燚)-云服务集团 wrote:

...

Thanks Brian, your explanation clarified something, but I don't get the answer if we can have multiple compute nodes are configured to dvr_snat, for this case, SNAT IPs are obviously different. Why do we want to use network node if compute node can do everything?

Hi Yi,

There will only be one DVR SNAT IP allocated for a router on the external network, and only one router scheduled using it, so having dvr_snat mode on a compute node doesn't mean that North/South router will be local, only the East/West portion might be.

Typically people choose to place these on separate systems since the requirements of the role are different - network node could have fewer cores and a 10G nic for higher bandwidth, compute node could have lots of cores for instances but maybe a 1G nic. There's no reason you can't run dvr_snat everywhere, I would just say it's not common.

-Brian

...

-----邮件原件----- 发件人: Brian Haley [mailto:haleyb.dev@gmail.com] 发送时间: 2019年5月16日 21:46 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: [DVR config] Can we use drv_snat agent_mode in every compute node?

Hi Yi,

I'm a little confused by the question, comments inline.

On 5/15/19 11:47 PM, Yi Yang (杨燚)-云服务集团 wrote:

...

Hi, folks

I saw somebody discussed distributed SNAT, but finally they didn’t make agreement on how to implement distributed SNAT, my question is can we use dvr_snat agent_mode in compute node? I understand dvr_snat only does snat but doesn’t do east west routing, right? Can we set dvr_snat and dvr in one compute node at the same time? It is equivalent to distributed SNAT if we can set drv_snat in every compute node, isn’t right? I know Opendaylight can do SNAT in compute node in distributed way, but one external router only can run in one compute node.

Distributed SNAT is not available in neutron, there was a spec proposed recently though, https://review.opendev.org/#/c/658414

Regarding the agent_mode setting for L3, only one mode can be set at a time. Typically 'dvr_snat' is used on network nodes and 'dvr' on compute nodes because it leads to less resource usage (i.e. namespaces). The centralized part of the router hosting the default SNAT IP address will only be scheduled to one of the agents in 'dvr_snat' mode. All the DVR modes can do East/West routing when an instance is scheduled to the node, and two can do North/South - 'dvr_snat' using the default SNAT IP, and 'dvr' using a floating IP. 'dvr_no_external' can only do East/West.

Hopefully that clarifies things.

-Brian

...

I also see https://wiki.openstack.org/wiki/Dragonflow is trying to implement distributed SNAT, what are technical road blocks for distributed SNAT in openstack dvr? Do we have any good way to remove these road blocks?

Thank you in advance and look forward to getting your replies and insights.

Also attached official drv configuration guide for your reference.

https://docs.openstack.org/neutron/stein/configuration/l3-agent.htm l

|agent_mode|¶ https://docs.openstack.org/neutron/stein/configuration/l3-agent.ht m l # DEFAULT.agent_mode

Type

string

Default

legacy

Valid Values

dvr, dvr_snat, legacy, dvr_no_external

The working mode for the agent. Allowed modes are: ‘legacy’ - this preserves the existing behavior where the L3 agent is deployed on a centralized networking node to provide L3 services like DNAT, and SNAT. Use this mode if you do not want to adopt DVR. ‘dvr’ - this mode enables DVR functionality and must be used for an L3 agent that runs on a compute host. ‘dvr_snat’ - this enables centralized SNAT support in conjunction with DVR. This mode must be used for an L3 agent running on a centralized node (or in single-host deployments, e.g. devstack). ‘dvr_no_external’ - this mode enables only East/West DVR routing functionality for a L3 agent that runs on a compute host, the North/South functionality such as DNAT and SNAT will be provided by the centralized network node that is running in ‘dvr_snat’ mode. This mode should be used when there is no external network connectivity on the compute host.

— Slawek Kaplonski Senior software engineer Red Hat

— Slawek Kaplonski Senior software engineer Red Hat