[tripleo] Dynamically generating firewall rules for the haproxy host on behalf of a deployed service
Hi all,
Designate has a requirement that appears to be a first for TripleO. It needs to deploy miniDNS instances on the internal network but each are accessible from the public network so external bind instances can sync with them. This is done by mapping port numbers on the VIP to each miniDNS instances [1] e.g.
10.0.0.5:16000 --> 172.16.2.184:5354 10.0.0.5:16001 --> 172.16.2.185:5354 10.0.0.5:16002 --> 172.16.2.186:5354 . . . <for each controller>
The haproxy configuration is relatively straightforward [2]. There would be no problem if it the haproxy puppet managed the firewall rules, but for typical deployments we appear to disable this [3] and rely on the tripleo.firewall ansible module instead.
The complicating factors are:
a. the ports range depends on the number of miniDNS instances.
b. the miniDNS deployment needs to set firewall rules wherever haproxy happens to live as they are not guaranteed to be on the same host. AFAICT, our ansible based firewall rule mechanism doesn't allow for setting firewall rules on hosts in addition to the ones that the service is being deployed on.
Possible solutions I've been considering:
a. Have haproxy puppet create firewall rules on the public API network and use tripleo.firewall for everything else. I think we generally don't distinguish which networks firewall rules are applicable to so this would be an "interesting", pervasive change. This is in PID 1's wheelhouse and I'd be interested in how they feel about this.
b. Somehow create a jinja-ed firewall rule for haproxy that gets processed at deploy time so details could be filled in using cloud information. This seems like it would be kind of fragile, may not even be possible and depends on details of the framework I'm not familiar with.
c. Extend tripleo.firewall or tripleo.iptables to process jinja'd rules - but that's problematic because it needs to be set on the haproxy host, and miniDNS is elsewhere.
d. Create some ansible that gets run in the haproxy hosts that generates the firewall rules based on cloud info. I feel this is a bit ugly.
Does anyone have any suggestions on other solutions or thoughts on the above?
Cheers,
Brent
1. https://bugzilla.redhat.com/show_bug.cgi?id=1897319 The TripleO HAProxy instance needs to be configured for BIND->miniDNS AXFR
2. https://review.opendev.org/c/openstack/puppet-tripleo/+/828169 Designate: create proxy ports on external VIP to access miniDNS workers.
3. https://bugs.launchpad.net/tripleo/+bug/1961799 haproxy template tasks to apply IPTables rules are no ops.
-- Brent Eagles Principal Software Engineer Red Hat Inc.
On Wed, Mar 2, 2022 at 8:59 AM Brent Eagles beagles@redhat.com wrote:
Hi all,
Designate has a requirement that appears to be a first for TripleO. It needs to deploy miniDNS instances on the internal network but each are accessible from the public network so external bind instances can sync with them. This is done by mapping port numbers on the VIP to each miniDNS instances [1] e.g.
10.0.0.5:16000 --> 172.16.2.184:5354 10.0.0.5:16001 --> 172.16.2.185:5354 10.0.0.5:16002 --> 172.16.2.186:5354 . . .
<for each controller>
The haproxy configuration is relatively straightforward [2]. There would be no problem if it the haproxy puppet managed the firewall rules, but for typical deployments we appear to disable this [3] and rely on the tripleo.firewall ansible module instead.
The complicating factors are:
a. the ports range depends on the number of miniDNS instances.
b. the miniDNS deployment needs to set firewall rules wherever haproxy happens to live as they are not guaranteed to be on the same host. AFAICT, our ansible based firewall rule mechanism doesn't allow for setting firewall rules on hosts in addition to the ones that the service is being deployed on.
Possible solutions I've been considering:
a. Have haproxy puppet create firewall rules on the public API network and use tripleo.firewall for everything else. I think we generally don't distinguish which networks firewall rules are applicable to so this would be an "interesting", pervasive change. This is in PID 1's wheelhouse and I'd be interested in how they feel about this.
This seems like the easiest path forward. That said, within the DF we're
actively attempting to reduce our dependence on puppet. As such, while the other options may be ugly or even more complex, I think they're worth pursuing especially if it means we have less puppet to maintain.
b. Somehow create a jinja-ed firewall rule for haproxy that gets processed at deploy time so details could be filled in using cloud information. This seems like it would be kind of fragile, may not even be possible and depends on details of the framework I'm not familiar with.
c. Extend tripleo.firewall or tripleo.iptables to process jinja'd rules
- but that's problematic because it needs to be set on the haproxy host, and miniDNS is elsewhere.
d. Create some ansible that gets run in the haproxy hosts that generates the firewall rules based on cloud info. I feel this is a bit ugly.
Does anyone have any suggestions on other solutions or thoughts on the above?
We make heavy use of YAQL to organize our firewall rules for
tripleo.firewall, is there something we can do there? We could also transition some of our firewall setup to use hostvars / facts across the deployment and to pull back information for specific nodes, or all nodes within a given group; maybe we can make better use of that within the firewall role to pull back runtime information and compute the needed firewall rules for minidns?
Cheers,
Brent
HAProxy instance needs to be configured for BIND->miniDNS AXFR
Designate: create proxy ports on external VIP to access miniDNS workers.
- https://bugs.launchpad.net/tripleo/+bug/1961799 haproxy template
tasks to apply IPTables rules are no ops.
-- Brent Eagles Principal Software Engineer Red Hat Inc.
I agree with Kevin. What I'd like to add here is that enabling haproxy firewall management by puppet means we have both ansible and puppet to manage firewall rules, and this is likely to cause undesired conjunction in the future. IMO it's better to find out the way to facilitate the requirement within tripleo-ansible(*1).
Note that this issue is not specific to Designate and can affect any deployment which has separate nodes for api servers and haproxy, as is mentioned in the bug.
I tend to think of a mixed approach of b and c, and gather frontend firewall rules for all services enabled in all roles by yaql, and then put that data into tripleo-firewall, then use that data only if haproxy is enabled in the node. I started drafting the change, but it's too late in my tz to implement such composition beyond roles ... https://review.opendev.org/q/topic:bug%252F1961799 (I hesitate to share this because it implements almost nothing but hope this explains my thought well)
(*1) By the way it reminds me that we still require puppet to set up NAT rules, which we also need to deal with, later, as a completely separate topic.
On Thu, Mar 3, 2022 at 12:43 AM Kevin Carter kecarter@redhat.com wrote:
On Wed, Mar 2, 2022 at 8:59 AM Brent Eagles beagles@redhat.com wrote:
Hi all,
Designate has a requirement that appears to be a first for TripleO. It needs to deploy miniDNS instances on the internal network but each are accessible from the public network so external bind instances can sync with them. This is done by mapping port numbers on the VIP to each miniDNS instances [1] e.g.
10.0.0.5:16000 --> 172.16.2.184:5354 10.0.0.5:16001 --> 172.16.2.185:5354 10.0.0.5:16002 --> 172.16.2.186:5354 . . .
<for each controller>
The haproxy configuration is relatively straightforward [2]. There would be no problem if it the haproxy puppet managed the firewall rules, but for typical deployments we appear to disable this [3] and rely on the tripleo.firewall ansible module instead.
The complicating factors are:
a. the ports range depends on the number of miniDNS instances.
b. the miniDNS deployment needs to set firewall rules wherever haproxy happens to live as they are not guaranteed to be on the same host. AFAICT, our ansible based firewall rule mechanism doesn't allow for setting firewall rules on hosts in addition to the ones that the service is being deployed on.
Possible solutions I've been considering:
a. Have haproxy puppet create firewall rules on the public API network and use tripleo.firewall for everything else. I think we generally don't distinguish which networks firewall rules are applicable to so this would be an "interesting", pervasive change. This is in PID 1's wheelhouse and I'd be interested in how they feel about this.
This seems like the easiest path forward. That said, within the DF we're
actively attempting to reduce our dependence on puppet. As such, while the other options may be ugly or even more complex, I think they're worth pursuing especially if it means we have less puppet to maintain.
b. Somehow create a jinja-ed firewall rule for haproxy that gets processed at deploy time so details could be filled in using cloud information. This seems like it would be kind of fragile, may not even be possible and depends on details of the framework I'm not familiar with.
c. Extend tripleo.firewall or tripleo.iptables to process jinja'd rules
- but that's problematic because it needs to be set on the haproxy host, and miniDNS is elsewhere.
d. Create some ansible that gets run in the haproxy hosts that generates the firewall rules based on cloud info. I feel this is a bit ugly.
Does anyone have any suggestions on other solutions or thoughts on the above?
We make heavy use of YAQL to organize our firewall rules for
tripleo.firewall, is there something we can do there? We could also transition some of our firewall setup to use hostvars / facts across the deployment and to pull back information for specific nodes, or all nodes within a given group; maybe we can make better use of that within the firewall role to pull back runtime information and compute the needed firewall rules for minidns?
Cheers,
Brent
HAProxy instance needs to be configured for BIND->miniDNS AXFR
Designate: create proxy ports on external VIP to access miniDNS workers.
- https://bugs.launchpad.net/tripleo/+bug/1961799 haproxy template
tasks to apply IPTables rules are no ops.
-- Brent Eagles Principal Software Engineer Red Hat Inc.
participants (3)
-
Brent Eagles
-
Kevin Carter
-
Takashi Kajinami