We are satisfied to announce the release of: octavia 10.1.0: OpenStack Octavia Scalable Load Balancer as a Service This release is part of the yoga stable release series. The source is available from: https://opendev.org/openstack/octavia Download the package from: https://pypi.org/project/octavia Please report issues through: https://storyboard.openstack.org/#!/project/908 For more details, please see below. 10.1.0 ^^^^^^ Known Issues ************ * When using a distribution with a recent SELinux release such as CentOS 8 Stream, PING health-monitor does not work as shell_exec_t calls are denied by SELinux. * Fixed configuration issue which allowed authenticated and authorized users to inject code into HAProxy configuration using API requests. Octavia API no longer accepts unencoded whitespace characters in url_path values in update requests for healthmonitors. Upgrade Notes ************* * The fix that updates the Netfilter Conntrack Sysfs variables requires rebuilding the amphora image in order to be effective. Security Issues *************** * Filter out private information from the taskflow logs when ''INFO'' level messages are enabled and when jobboard is enabled. Logs might have included TLS certificates and private_key. By default, in Octavia only WARNING and above messages are enabled in taskflow and jobboard is disabled. Bug Fixes ********* * In order to avoid hitting the Neutron API hard when batch update with creating many new members, we cache the subnet validation results in batch update members API call. We also change to validate new members only during batch update members since subnet ID is immutable. * Ensure that the provided rsyslog configuration file is used by rsyslog in the amphora by restarting the service when using the amphorav1 provider, it fixes the log offloading feature on distributions that start rsyslog before cloud-init. * The parameters of a taskflow Flow were logged in ''INFO'' level messages by taskflow, it included TLS-enabled listeners and pools parameters, such as certificates and private_key. * Fix an authentication error with Barbican when creating a TERMINATED_HTTPS listener with application credential tokens or trust IDs. * Fixed a potential race condition in the member batch update API call, the load balancers might not have been locked properly. * Fix an issue that may have occurred when running the amphorav2 with persistence, the ComputeActiveWait was incorrectly executed twice on different controllers. * Fixed a "corrupted global server state file" error in Centos 9 Stream when reloading the state of the servers after restarting haproxy. It also fixed the recovering of the operational state of the servers in haproxy after its restart. * Fix disabled UDP pools. Disabled UDP pools were marked as "OFFLINE" but the requests were still forwarded to the members of the pool. * Fix the shutdown of the driver-agent, the process might have been stuck while waiting for threads to finish. Systemd would have killed the process after a timeout, but some children processes might have leaked on the controllers. * Enable required SELinux booleans for CentOS or RHEL amphora image. * Fix a bug when full graph of load balancer is created without listeners if jobboard_enabled=False * Fixed a bug that prevented Octavia from creating listeners with the fully-populated load balancer API in SINGLE topology mode. * Fixed backwards compatibility issue with the feature that preserves HAProxy server states between reloads. HAProxy version 1.5 or below do not support this feature, so Octavia will not to activate it on amphorae with those versions. * Fix a bug that prevented the provisioning_state of a health- monitor to be set to ERROR when an error occurred while creating, updating or deleting a health-monitor. * Fixed a bug that didn't set all the active load balancer Health Monitors ONLINE in populated LB single-create calls. * Fix a bug that prevented the operating_status of a health-monitor to be set to ONLINE when ipv6 addresses were enclosed within square brackets in "controller_ip_port_list". * Fixes listener creation failure when protocol used is PROXY or PROXYV2 which are pool protocol and not listener protocol. * Fix update listener certs failed. The fix ensures that an existing certificate gets overwritten properly. * Fixed a potential error when plugging a member from a new network after deleting another member and unplugging its network. Octavia may have tried to plug the new network to a new interface but with an already existing name. This fix requires to update the Amphora image. * Netfilter Conntrack Sysfs variables net.netfilter.nf_conntrack_max and nf_conntrack_expect_max get set to sensible values on the amphora now. Previously, kernel default values were used which were much too low for the configured net.netfilter.nf_conntrack_buckets value. As a result packets could get dropped because the conntrack table got filled too quickly. Note that this affects only UDP and SCTP protocol listeners. Connection tracking is disabled for TCP- based connections on the amphora including HTTP(S). * Now the "[nova] service_name" parameter is effectively used to find the nova endpoint in keystone catalog. The parameter had no effect before it was fixed. * Fix an issue with PING health-monitors on Centos 8 Stream. Changes in Centos and systemd prevent an unprivileged user from sending ping requests from a network namespace. * Fix PING health-monitors with recent haproxy releases (>=2.2), haproxy now requires an additional "insecure-fork-wanted" option to authorize the Octavia PING healthcheck. * Fix a bug when adding a member on a subnet that belongs to a network with multiple subnets, an incorrect subnet may have been plugged in the amphora. * Fix a bug when deleting the last member plugged on a network, the port that was no longer used was not deleted. * Fixed a bug that didn't set the correct provisioning_status for unattached pools when creating a fully-populated load balancer. * Fix a bug when updating a load balancer with a QoS policy after a failover, Octavia attempted to update the VRRP ports of the deleted amphorae, moving the provisioning status of the load balancer to ERROR. * Fix a potential race condition when updating a resource in the amphorav2 worker. The worker was not waiting for the resource to be set to PENDING_UPDATE, so the resource may have been updated with old data from the database, resulting in a no-op update. * Fix the rescheduling of taskflow tasks that have been resumed after being interrupted. * Fixed issue with SELinux and the lvs-masquerade.sh script on the amphora. The script already runs with root permissions, so the use of sudo inside the script is unneeded. * Fixed an SELinux issues with TCP-based health-monitor on UDP pools, some specific monitoring ports were denied by SELinux. The Amphora image now enables the "keepalived_connect_any" SELinux boolean that allows connections to any ports. * Fix an issue when Octavia performs a failover of an ACTIVE-STANDBY load balancer that has both amphorae missing. Some tasks in the controller took too much time to timeout because the timeout value defined in "[haproxy_amphora].active_connection_max_retries" and "[haproxy_amphora].active_connection_rety_interval" was not used. * Fix a serialization issue when using TLSContainer with amphorav2 driver with persistence, a list of bytes type in the data model was not correctly converted to serializable data. * Fixed "Could not retrieve certificate" error when updating/deleting the client_ca_tls_container_ref field of a listener after a CA/CRL was deleted. * Fix a python3 error that prevented to use the "[controller_worker]/user_data_config_drive" option when building amphorae. * Fixed validations in L7 rule and session cookie APIs in order to prevent authenticated and authorized users to inject code into HAProxy configuration. CR and LF (\r and \n) are no longer allowed in L7 rule keys and values. The session persistence cookie names must follow the rules described in https://developer.mozilla.org/en- US/docs/Web/HTTP/Headers/Set-Cookie. * Fix load balancers stuck in PENDING_UPDATE issues for some API calls (POST /l7rule, PUT /pool) when a provider denied the call. * The Octavia API returned an unhelpful message when a constraint failed while creating an object in the DB. The error now contains the name and the value of the parameter that breaks the constraints. * When plugging a new member subnet, the amphora sends an IP advertisement of the newly allocated IP. It allows the servers on the same L2 network to flush the ARP entries of a previously allocated IP address. * Validate that the creation of L7 policies is compatible with the protocol of the listener in the Amphora driver. L7 policies are allowed for Terminated HTTPS or HTTP protocol listeners, but not for HTTPS, TCP or UDP protocols listeners. Other Notes *********** * The string representation of data base model objects has been improved. Calling str() on them will return a certain subset of fields and calling repr() on them will return all fields. This is helpful for debugging, but it may also change some of the log messages that Octavia emits. Changes in octavia 10.0.0..10.1.0 --------------------------------- da9dc123 Fix TCP HMs on UDP pools with SELinux b555ffc2 Fix hm operating status to ONLINE in single lb call 202d1dc4 Avoid interface name collisions in the amphora dabd33af Fix pool creation with single LB create call e0f7a145 Send IP advertisements when plugging a new member subnet 09e0beb6 Fix pep8 error dd29742d Prevent incorrect reschedule of resumed tasks with jobboard db22c0b9 Fix serialization of lists in data_models.to_dict e4cfb3e5 Remove incorrect info message ae3cc5b3 Enable taskflow retry feature when waiting for compute 0c8c5037 Fix octavia to accept [ipv6]:port df2a5275 Fix ORM caching for with_for_update calls cc276721 Fix ignored [nova] service_name 6ae3d23e Fix listener creation allowing pool protocol d3191eb9 Fix unclear error messages in the API 524ce505 Fix update listener certs doesn't work 3b3458b7 Fix template issue with user_data_config_drive 3380b292 Fix prometheus-proxy systemd service 4b588d4e Fix grenade job 23a7ca77 Pin pylint on stable/branches 46f774ad Filter out details from taskflow logs with v2+jobboard 2e5bc334 Fix PING health-monitor with recent Centos releases d4d8d597 Fix PING health-monitor with recent haproxy releases 66a62bc2 Fix listener creation with fully-populated API c662e42b Fix full graph loadbalancer creation if jobboard is disabled 3605689a Add a newline when writing the server state file 8aa5eaa8 Handle feature compatibility of HAProxy server-state-file option 74736870 Fix barbican client with application credentials/trusts 5efaf360 Fix prometheus-proxy service name in Red Hat-based distros 0d8a06bd Use python3.9 for tips jobs d4b0c4f5 Restart rsyslog from cloud-init in amphorav1 10042f76 Add *.orig to .gitignore 64947c5f Fix sporadic unit test failure 62e1d1fa Update zuul queue configuration 71c61b80 Fix plugging member subnets on existing networks e76efd82 Passphrase parameters should be secret d75613fa Reconfigure amphora network interfaces seamlessly 0598c4fb Ignore status update on deleted objects in driver-agent 384619ef Cache subnets validation for batch member update 3e6f180b Fix bug when rolling back prov and op status for some API calls 328ffc00 Fix PortNotFound exception when updating a LB after a failover d5565417 Catch exceptions on I/O in driver-agent 995fe216 Apply openstack-selinux policies in Centos amphorae ef76cbf7 Change FIPS jobs to centos-9-stream 1a1c5d42 Fix driver-agent cleanup 74f2baf5 Fix update/delete listener CA/CRL error c8dd836e Move system scoped secure-RBAC to separate file 5ca3849a Fix HealthMonitorToErrorOnRevertTask revert method 2cb52fe6 Improve string representation of DB models e6989982 Fix duplicate object error messages 9a25da98 Fix potential race conditions on update requests in the v2 worker 78603681 Set sensible nf_conntrack_max value in amphora 7c3f2a57 Exclude invalid TLS version ef4ef042 Correct format of release note a2371823 Fix compile_amphora_details when using UDP listeners e397a118 Fix disabled UDP pools 133ec476 Validate L7Rule value and cookie name 399c2f17 Fix new pylint issues 9afb3af6 Reject invalid whitespace in HM url_path value 0df7c6b7 Remove unneeded sudo in lvs-masquerade.sh 4248e98a Deny the creation of L7Policies for HTTPS/TCP/UDP listeners 7aaac224 Fix AttributeError in exception handler 0a062cd6 Save the HAProxy state outside of its systemd unit 1da04d12 Pass timeout_dict to _get_haproxy_versions a8a35e08 Update TOX_CONSTRAINTS_FILE for stable/yoga c2939bc6 Update .gitreview for stable/yoga Diffstat (except docs and test files) ------------------------------------- .gitignore | 2 + .gitreview | 1 + .pylintrc | 5 - diskimage-create/diskimage-create.sh | 9 +- .../prometheus-proxy.service | 2 +- .../12-enable-prometheus-proxy-systemd | 2 +- .../amphora-agent/source-repository-amphora-agent | 4 +- .../static/usr/local/bin/lvs-masquerade.sh | 4 +- elements/amphora-agent/svc-map | 3 + elements/amphora-selinux/README.rst | 3 + elements/amphora-selinux/element-deps | 2 + elements/amphora-selinux/package-installs.json | 4 + elements/amphora-selinux/pkg-map | 12 + .../post-install.d/50-selinux-policies | 22 + .../post-install.d/20-haproxy-tune-kernel | 8 + elements/octavia-lib/source-repository-octavia-lib | 2 +- etc/policy/README.rst | 12 + etc/policy/keystone_default_roles-policy.yaml | 6 +- .../keystone_default_roles_scoped-policy.yaml | 37 + .../backends/agent/api_server/amphora_info.py | 8 +- .../backends/agent/api_server/loadbalancer.py | 14 +- .../amphorae/backends/agent/api_server/osutils.py | 26 +- octavia/amphorae/backends/agent/api_server/plug.py | 103 ++- .../amphorae/backends/agent/api_server/server.py | 3 +- .../api_server/templates/amphora-netns.systemd.j2 | 3 + .../agent/api_server/templates/systemd.conf.j2 | 1 - octavia/amphorae/backends/agent/api_server/util.py | 21 + .../backends/health_daemon/health_sender.py | 2 + octavia/amphorae/backends/utils/haproxy_query.py | 28 +- octavia/amphorae/backends/utils/interface.py | 231 +++++- octavia/amphorae/backends/utils/interface_file.py | 106 ++- octavia/amphorae/backends/utils/network_utils.py | 3 +- octavia/amphorae/drivers/driver_base.py | 9 +- .../amphorae/drivers/haproxy/rest_api_driver.py | 104 ++- .../drivers/keepalived/vrrp_rest_driver.py | 3 +- octavia/amphorae/drivers/noop_driver/driver.py | 11 +- octavia/api/drivers/amphora_driver/v1/driver.py | 13 + octavia/api/drivers/amphora_driver/v2/driver.py | 13 + .../api/drivers/driver_agent/driver_listener.py | 99 +-- octavia/api/drivers/driver_agent/driver_updater.py | 14 +- octavia/api/drivers/utils.py | 12 +- octavia/api/v2/controllers/health_monitor.py | 8 +- octavia/api/v2/controllers/l7policy.py | 8 +- octavia/api/v2/controllers/l7rule.py | 10 +- octavia/api/v2/controllers/listener.py | 14 +- octavia/api/v2/controllers/member.py | 48 +- octavia/api/v2/controllers/pool.py | 10 +- octavia/api/v2/types/l7rule.py | 13 +- octavia/api/v2/types/listener.py | 7 +- octavia/api/v2/types/pool.py | 14 +- octavia/certificates/common/auth/barbican_acl.py | 27 +- octavia/certificates/common/local.py | 6 +- octavia/common/base_taskflow.py | 97 ++- octavia/common/config.py | 8 +- octavia/common/constants.py | 16 +- octavia/common/data_models.py | 95 +-- .../jinja/haproxy/combined_listeners/jinja_cfg.py | 18 +- .../haproxy/combined_listeners/templates/base.j2 | 5 + .../combined_listeners/templates/haproxy.cfg.j2 | 2 +- .../haproxy/combined_listeners/templates/macros.j2 | 4 +- octavia/common/jinja/lvs/templates/macros.j2 | 6 +- octavia/common/validate.py | 1 + octavia/compute/drivers/nova_driver.py | 1 + octavia/controller/worker/v1/controller_worker.py | 3 + .../controller/worker/v1/flows/amphora_flows.py | 7 +- .../controller/worker/v1/flows/listener_flows.py | 3 + .../worker/v1/flows/load_balancer_flows.py | 33 +- octavia/controller/worker/v1/flows/member_flows.py | 31 +- .../worker/v1/tasks/amphora_driver_tasks.py | 14 +- .../controller/worker/v1/tasks/compute_tasks.py | 9 +- .../controller/worker/v1/tasks/database_tasks.py | 44 +- .../controller/worker/v1/tasks/lifecycle_tasks.py | 2 +- .../controller/worker/v1/tasks/network_tasks.py | 257 +++++-- octavia/controller/worker/v2/controller_worker.py | 94 ++- .../controller/worker/v2/flows/amphora_flows.py | 36 +- .../controller/worker/v2/flows/listener_flows.py | 3 + .../worker/v2/flows/load_balancer_flows.py | 34 +- octavia/controller/worker/v2/flows/member_flows.py | 31 +- .../worker/v2/tasks/amphora_driver_tasks.py | 16 +- .../controller/worker/v2/tasks/compute_tasks.py | 43 +- .../controller/worker/v2/tasks/database_tasks.py | 45 +- .../controller/worker/v2/tasks/lifecycle_tasks.py | 2 +- .../controller/worker/v2/tasks/network_tasks.py | 258 +++++-- octavia/db/base_models.py | 7 + octavia/db/models.py | 63 ++ octavia/db/repositories.py | 94 ++- octavia/hacking/checks.py | 2 +- octavia/network/base.py | 31 +- octavia/network/data_models.py | 5 +- .../drivers/neutron/allowed_address_pairs.py | 10 +- octavia/network/drivers/neutron/base.py | 32 + octavia/network/drivers/noop_driver/driver.py | 76 +- .../backend/agent/api_server/test_server.py | 74 +- .../api/drivers/driver_agent/test_driver_agent.py | 37 + .../backends/agent/api_server/test_amphora_info.py | 13 +- .../agent/api_server/test_haproxy_compatibility.py | 2 - .../backends/agent/api_server/test_loadbalancer.py | 5 +- .../backends/agent/api_server/test_osutils.py | 16 + .../backends/agent/api_server/test_plug.py | 200 ++++- .../backends/agent/api_server/test_util.py | 35 + .../backends/health_daemon/test_health_sender.py | 18 + .../amphorae/backends/utils/test_haproxy_query.py | 28 + .../unit/amphorae/backends/utils/test_interface.py | 282 ++++++- .../amphorae/backends/utils/test_interface_file.py | 160 +++- .../drivers/haproxy/test_rest_api_driver_0_5.py | 60 +- .../drivers/haproxy/test_rest_api_driver_1_0.py | 90 ++- .../amphorae/drivers/noop_driver/test_driver.py | 4 +- .../api/drivers/amphora_driver/v1/test_driver.py | 20 +- .../api/drivers/amphora_driver/v2/test_driver.py | 20 +- .../drivers/driver_agent/test_driver_listener.py | 173 ++++- .../certificates/common/auth/test_barbican_acl.py | 3 +- .../haproxy/combined_listeners/test_jinja_cfg.py | 186 ++++- .../sample_configs/sample_configs_combined.py | 26 +- .../common/sample_configs/sample_configs_split.py | 20 +- .../worker/v1/flows/test_amphora_flows.py | 8 +- .../worker/v1/flows/test_load_balancer_flows.py | 19 +- .../worker/v1/flows/test_member_flows.py | 25 +- .../worker/v1/tasks/test_amphora_driver_tasks.py | 18 +- .../worker/v1/tasks/test_compute_tasks.py | 44 +- .../worker/v1/tasks/test_database_tasks.py | 63 +- .../worker/v1/tasks/test_lifecycle_tasks.py | 2 +- .../worker/v1/tasks/test_network_tasks.py | 835 ++++++++++++++++++--- .../controller/worker/v1/test_controller_worker.py | 5 + .../worker/v2/flows/test_amphora_flows.py | 4 +- .../worker/v2/flows/test_load_balancer_flows.py | 19 +- .../worker/v2/flows/test_member_flows.py | 21 +- .../worker/v2/tasks/test_amphora_driver_tasks.py | 26 +- .../worker/v2/tasks/test_compute_tasks.py | 8 +- .../worker/v2/tasks/test_database_tasks.py | 56 +- .../worker/v2/tasks/test_lifecycle_tasks.py | 2 +- .../worker/v2/tasks/test_network_tasks.py | 803 ++++++++++++++++---- .../controller/worker/v2/test_controller_worker.py | 270 ++++++- .../drivers/neutron/test_allowed_address_pairs.py | 7 +- .../unit/network/drivers/neutron/test_base.py | 141 ++++ .../network/drivers/noop_driver/test_driver.py | 67 +- .../notes/catch_validation-27ffe48ca187c46f.yaml | 8 + ...syslog-reloaded-amphorav1-a4ec5127a459f3bf.yaml | 7 + ...mation-from-taskflow-logs-0d8697140423b4d5.yaml | 12 + ...tial-tokens-with-barbican-3b7d13283206c124.yaml | 5 + ...ber-update-race-condition-09b82e2cc3121e03.yaml | 5 + ...putewait-with-persistence-d10223bfb48a0ded.yaml | 6 + ...-global-server-state-file-325ab7c62e21ff14.yaml | 7 + .../fix-disabled-udp-pool-3e84558f996017d5.yaml | 5 + ...r-agent-graceful-shutdown-daff9ffaccb09a9e.yaml | 7 + ...nforced-selinux-on-centos-27842ca6afbb500c.yaml | 4 + ...dbalancer-creation-if-jobboard-is-disabled.yaml | 5 + ...opulated-lb-with-listener-92a369ea8d57e8f5.yaml | 5 + ...y-about-server-state-file-df70e5ac859417e2.yaml | 7 + ...itor-to-error-revert-task-feb38ba7641a4892.yaml | 6 + ...-online-in-single-lb-call-214a7ca22937a877.yaml | 5 + ...ress-enclosed-in-brackets-c1cfc4717465ba09.yaml | 6 + ...on-allowing-pool-protocol-b9e9ef147f6eeaf4.yaml | 5 + ...tener-update-certs-failed-315c66f4806e76c8.yaml | 5 + ...twork-interface-collision-939fd32587ea3344.yaml | 8 + ...onntrack-max-value-in-amp-0e16eb50b42e7b58.yaml | 15 + .../fix-nova-service_name-6bde4970047817f4.yaml | 6 + ...-ping-hm-on-centos-stream-6624f19c8da86e22.yaml | 6 + ...-ping-hm-with-haproxy-2.2-9b83777172fb8835.yaml | 6 + ...x-plugging-member-subnets-8560cd9403ff79a7.yaml | 8 + ...tatus-on-lb-single-create-897070aee0a42da6.yaml | 5 + ...-qos-apply-after-failover-561abbd153ab88ee.yaml | 6 + ...race-condiction-on-update-b5330c8fcf1800cd.yaml | 7 + ...chedule-of-jobboard-tasks-929c066dea9267fd.yaml | 5 + ...ue-with-lvs-masquerade.sh-ebbb89886148c70f.yaml | 6 + ...linux-tcp-hm-on-udp-pools-89c3b8db89e359ba.yaml | 7 + ...ut-dict-in-failover-tasks-537456e0fe1d7cb8.yaml | 9 + ...s-container-serialization-1cb83ad4c9eca3b8.yaml | 6 + ...-update-listener-ca-error-167464debc06cba2.yaml | 5 + ...ix-user_data_config_drive-b4ce8cc66fb21365.yaml | 6 + ...rules-and-session-cookies-cb88f3f1b90171f9.yaml | 9 + ...UPDATE-on-provider-errors-40a03adc8ef82a54.yaml | 5 + ...essage-failed-constraints-6ad10bd22cac523a.yaml | 6 + ...epresentation-of-db-model-1c4fe799186b4dea.yaml | 7 + ...-subnet-ip-advertisements-af2264844079ef6b.yaml | 6 + .../ping-healthcheck-selinux-e3b7d360c8503527.yaml | 6 + ...-protocols-for-l7policies-83d678171f13136a.yaml | 7 + ...rl_path-value-in-requests-3eb3adedcd696433.yaml | 7 + test-requirements.txt | 2 +- tox.ini | 8 +- zuul.d/jobs.yaml | 45 +- zuul.d/projects.yaml | 2 +- 197 files changed, 6080 insertions(+), 1231 deletions(-) Requirements updates -------------------- diff --git a/test-requirements.txt b/test-requirements.txt index a4e7a9ba..555d738c 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -11 +11 @@ oslotest>=3.2.0 # Apache-2.0 -pylint>=2.5.3 # GPLv2 +pylint>=2.5.3,<=2.15.10 # GPLv2