- openstack-discuss - lists.openstack.org

[neutron] Bug deputy Jan 5 - Jan 11
by Rodolfo Alonso Hernandez 12 Jan '26

12 Jan '26

Hello Neutrinos: This is the list of bugs created during the last week: * Critical ** networking-bgpvpn and neutron-vpnaas unit tests arefailing due to tenant_id to project_id migration ( https://bugs.launchpad.net/neutron/+bug/2137489) Fixes merged and bug closed. * High: ** devstack deployment failing on stable/2025.1 branch without wsgi ( https://bugs.launchpad.net/neutron/+bug/2137540) Fix proposed: https://review.opendev.org/c/openstack/neutron/+/972669 (tested in https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/972266) * Medium ** neutron.tests.functional.services.bgp.test_ovn.TestBgpOvnLocking.test_locking_mechanism randomly fails with AssertionError: Second API did not lose the lock ( https://bugs.launchpad.net/neutron/+bug/2137467) In progress: https://review.opendev.org/c/openstack/neutron/+/972547 ** ``get_api_worker_id`` should not fail if ``uwsgi`` module is not installed (https://bugs.launchpad.net/neutron/+bug/2137715) Fix proposed: https://review.opendev.org/c/openstack/neutron/+/972687 (needed up to 2025.2) Regards.

1 0

[tacker] Cancelling weekly IRC meeting
by Yasufumi Ogawa 11 Jan '26

11 Jan '26

Hi tacker team, Due to a holiday in Japan and most of team members aren't join, I'm cancelling meeting this week. Regards, Yasufumi

1 0

[release] Release countdown for week R-11, Jan 12 - 16
by Elõd Illés 09 Jan '26

09 Jan '26

Development Focus ----------------- We are now past the Gazpacho-2 milestone, and entering the last development phase of the cycle. Please review the late milestone-2 release patches at https://review.opendev.org/q/topic:gazpacho-milestone-2 if you have not done so already! Teams should otherwise be focused on implementing planned work for the cycle. Now is a good time to review those plans and reprioritize anything if needed based on the what progress has been made and what looks realistic to complete in the next few weeks. General Information ------------------- Looking ahead to the end of the release cycle, please be aware of the feature freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (February 19th, 2026). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (February 26th, 2026) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, February 26th, 2026. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (March 12th, 2026) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (March 26th, 2026) Upcoming Deadlines & Dates -------------------------- Non-client library freeze: February 19th, 2026 (R-6 week) Client library freeze: February 26th, 2026 (R-5 week) Gazpacho-3 milestone: February 26th, 2026 (R-5 week) Előd Illés irc: elodilles @ #openstack-release

1 0

[Nova] Proposal: NetApp FlexGroup Backend for Ceph-like Ephemeral Storage Performance on NFS
by Borra, Jayaanand 08 Jan '26

08 Jan '26

Hi everyone, Happy New Year to all! I hope you all had a great holiday season. I'm reaching out to share a proposal we've been working on that addresses a performance challenge many enterprises face with ephemeral storage in Nova. Context: Currently, OpenStack deployments running on NFS-based storage (like NetApp FlexGroups) face a significant performance penalty compared to Ceph RBD due to the QCOW2-over-NFS approach. The hypervisor manages Copy-on-Write operations, which introduces: * Double filesystem metadata overhead (QCOW2 on top of NFS) * CPU tax on compute nodes for CoW management * Slower instance provisioning compared to Ceph's instant cloning We are planning to implement solution that leverages NetApp's FlexClone technology through the ONTAP REST API to achieve Ceph-like performance while maintaining the operational simplicity of NFS. Essentially, we're offloading the cloning operation to the storage array itself, making instance creation a metadata-only operation (sub-second) with zero data copying. The implementation involves creating a new NetAppNfs image backend class in nova/virt/libvirt/imagebackend.py that calls the ONTAP REST API's /storage/file/clone endpoint instead of traditional file copy operations. I'm wondering if anyone has insights into: * Similar efforts: Has anyone explored or implemented storage-side cloning for other NFS vendors? * Design concerns: Any architectural red flags with this approach in the Nova driver? * Security practices: Best approach for managing storage credentials and oslo.privsep<vscode-file://vscode-app/Applications/Visual%20Studio%20Code.app/Contents/Resources/app/out/vs/code/electron-browser/workbench/workbench.html> integration? * Community interest: Would this be valuable for other NFS deployments? This will enable high-performance ephemeral storage from, so I'm curious if others have encountered this limitation and whether there's are any known limitation with this approach. Thanks in advance for any guidance or suggestions! Best regards, Jayaanand.

1 0

SECURITY BUG
by Umbrella Corporation 08 Jan '26

08 Jan '26

i found a vulnerability in your git repo so if i reported bugs and vulnerabilities can i get a reward bounty ?

3 3

OpenStack 2026.2 'H' Release Name Poll
by Allison Price 08 Jan '26

08 Jan '26

Hi everyone, Happy New Year! To kick off 2026, I wanted to share the poll to vote for the OpenStack 2026.2 release name, starting with the letter H. The poll will close this Thursday, January 8 at 1700 UTC. https://civs1.civs.us/cgi-bin/vote.pl?id=E_e4708c67d8d7e171&akey=e3b1514078… If you have any questions / comments, please let me know. Cheers, Allison

1 1

[designate] Question about NS record update restriction in multi-region PRIMARY/SECONDARY setup
by Sylvain 08 Jan '26

08 Jan '26

Hi everyone, I hope this email finds you well. I'm reaching out to the community because I'm encountering a limitation with Designate that I don't fully understand, and I'm hoping someone can shed some light on the reasoning behind it or suggest alternative approaches. My Setup I'm running a multi-region Designate deployment with: - 3 independent Designate instances (one per region) - Each instance has its own pool with a single nameserver: - Region 1: ns.region1.net. - Region 2: ns.region2.net. - Region 3: ns.region3.net. This default configuration works perfectly for most of our use cases where zones are region-specific. What I'm Trying to Achieve For certain domains, I need to set up a PRIMARY/SECONDARY configuration across all three regions: - Region 1: PRIMARY zone for example.com - Region 2: SECONDARY zone for example.com - Region 3: SECONDARY zone for example.com For this to work correctly, the PRIMARY zone needs to have NS records pointing to all three nameservers so that: 1. DNS clients can query any of the three nameservers 2. Zone transfers are properly configured between primary and secondaries The Problem When I try to update the NS recordset on the primary zone, I get the error: Updating a root zone NS record is not allowed Steps I'm following: # List recordsets to get the NS recordset ID openstack recordset list example.com. # Attempt to update the NS recordset openstack recordset set example.com. <ns-recordset-id> \ --record ns.region1.net. \ --record ns.region2.net. \ --record ns.region3.net. This happens even when authenticated as an admin user. Why the Default Workaround Doesn't Work I understand that one typical solution is to modify the pool configuration to include all three nameservers. However, this doesn't work in my case because: - I don't want to change the default pool settings since most zones should only use their local regional nameserver - Modifying the pool in Region 1 would affect *all zones* created there, not just the ones I want to replicate My Questions I'm trying to understand the design decisions here, and I'd really appreciate any insights: 1. What's the reasoning behind blocking NS record updates on root zones? I'm not seeing the security or technical concern that this protects against. 2. Is this restriction something that could potentially be made configurable via policy? For example, allowing zone owners to update their own NS records when they have legitimate multi-region or high-availability requirements. 3. Are there any alternative approaches I should be considering for this multi-region PRIMARY/SECONDARY setup that I might be missing? 4. Would the community be open to discussing a potential enhancement that makes this more flexible? I understand if there are concerns, but I'd love to understand them better. Use Case Context This is a fairly standard DNS architecture for high availability and geographic distribution. Many organizations need to run primary and secondary nameservers across different regions or datacenters, and being able to configure the NS records accordingly seems like it should be a supported use case. Thank you very much for taking the time to read this, and I appreciate any guidance or feedback the community can provide! Best regards, Sylvain.

2 2

vmware_nsxv
by Parsa Aminian 07 Jan '26

07 Jan '26

hello on kolla ansible neutron_plugin_agent: "vmware_nsxv" is showing this error : fatal: [control01]: FAILED! => {"changed": true, "msg": "Container exited with non-zero return code 1", "rc": 1, "stderr": "+ sudo -E kolla_set_configs\n2026-01-06 12:53:23.464 INFO Loading config file at /var/lib/kolla/config_files/config.json\n2026-01-06 12:53:23.465 INFO Validating config file\n2026-01-06 12:53:23.465 INFO Kolla config strategy set to: COPY_ALWAYS\n2026-01-06 12:53:23.476 INFO Copying service configuration files\n2026-01-06 12:53:23.476 INFO Copying /var/lib/kolla/config_files/neutron.conf to /etc/neutron/neutron.conf\n2026-01-06 12:53:23.489 INFO Setting permission for /etc/neutron/neutron.conf\n2026-01-06 12:53:23.501 INFO Copying /var/lib/kolla/config_files/nsx.ini to /etc/neutron/plugins/vmware/nsx.ini\n2026-01-06 12:53:23.501 INFO Setting permission for /etc/neutron/plugins/vmware/nsx.ini\n2026-01-06 12:53:23.502 INFO Copying /var/lib/kolla/config_files/ml2_conf.ini to /etc/neutron/plugins/ml2/ml2_conf.ini\n2026-01-06 12:53:23.512 INFO Setting permission for /etc/neutron/plugins/ml2/ml2_conf.ini\n2026-01-06 12:53:23.523 INFO Copying /var/lib/kolla/config_files/id_rsa to /var/lib/neutron/.ssh/id_rsa\n2026-01-06 12:53:23.524 INFO Setting permission for /var/lib/neutron/.ssh/id_rsa\n2026-01-06 12:53:23.524 INFO Writing out command to execute\n++ cat /run_command\n+ CMD='neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/vmware/nsx.ini '\n+ ARGS=\n+ sudo kolla_copy_cacerts\n+ sudo kolla_install_projects\n+ [[ ! -n '' ]]\n+ . kolla_extend_start\n++ [[ ! -d /var/log/kolla/neutron ]]\n++ mkdir -p /var/log/kolla/neutron\n+++ stat -c %a /var/log/kolla/neutron\n++ [[ 2755 != \\7\\5\\5 ]]\n++ chmod 755 /var/log/kolla/neutron\n++ [[ ! ubuntu =~ centos|rocky ]]\n++ /usr/bin/update-alternatives --display iptables\n++ KOLLA_LEGACY_IPTABLES=false\n++ [[ false == \\t\\r\\u\\e ]]\n++ sudo /usr/bin/update-alternatives --auto iptables\n++ sudo /usr/bin/update-alternatives --auto ip6tables\n++ . /usr/local/bin/kolla_neutron_extend_start\n+++ [[ -n 0 ]]\n+++ neutron-db-manage --subproject neutron upgrade head\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO [alembic.runtime.migration] Running upgrade -> kilo\nINFO [alembic.runtime.migration] Running upgrade kilo -> 354db87e3225\nINFO [alembic.runtime.migration] Running upgrade 354db87e3225 -> 599c6a226151\nINFO [alembic.runtime.migration] Running upgrade 599c6a226151 -> 52c5312f6baf\nINFO [alembic.runtime.migration] Running upgrade 52c5312f6baf -> 313373c0ffee\nINFO [alembic.runtime.migration] Running upgrade 313373c0ffee -> 8675309a5c4f\nINFO [alembic.runtime.migration] Running upgrade 8675309a5c4f -> 45f955889773\nINFO [alembic.runtime.migration] Running upgrade 45f955889773 -> 26c371498592\nINFO [alembic.runtime.migration] Running upgrade 26c371498592 -> 1c844d1677f7\nINFO [alembic.runtime.migration] Running upgrade 1c844d1677f7 -> 1b4c6e320f79\nINFO [alembic.runtime.migration] Running upgrade 1b4c6e320f79 -> 48153cb5f051\nINFO [alembic.runtime.migration] Running upgrade 48153cb5f051 -> 9859ac9c136\nINFO [alembic.runtime.migration] Running upgrade 9859ac9c136 -> 34af2b5c5a59\nINFO [alembic.runtime.migration] Running upgrade 34af2b5c5a59 -> 59cb5b6cf4d\nINFO [alembic.runtime.migration] Running upgrade 59cb5b6cf4d -> 13cfb89f881a\nINFO [alembic.runtime.migration] Running upgrade 13cfb89f881a -> 32e5974ada25\nINFO [alembic.runtime.migration] Running upgrade 32e5974ada25 -> ec7fcfbf72ee\nINFO [alembic.runtime.migration] Running upgrade ec7fcfbf72ee -> dce3ec7a25c9\nINFO [alembic.runtime.migration] Running upgrade dce3ec7a25c9 -> c3a73f615e4\nINFO [alembic.runtime.migration] Running upgrade c3a73f615e4 -> 659bf3d90664\nINFO [alembic.runtime.migration] Running upgrade 659bf3d90664 -> 1df244e556f5\nINFO [alembic.runtime.migration] Running upgrade 1df244e556f5 -> 19f26505c74f\nINFO [alembic.runtime.migration] Running upgrade 19f26505c74f -> 15be73214821\nINFO [alembic.runtime.migration] Running upgrade 15be73214821 -> b4caf27aae4\nINFO [alembic.runtime.migration] Running upgrade b4caf27aae4 -> 15e43b934f81\nINFO [alembic.runtime.migration] Running upgrade 15e43b934f81 -> 31ed664953e6\nINFO [alembic.runtime.migration] Running upgrade 31ed664953e6 -> 2f9e956e7532\nINFO [alembic.runtime.migration] Running upgrade 2f9e956e7532 -> 3894bccad37f\nINFO [alembic.runtime.migration] Running upgrade 3894bccad37f -> 0e66c5227a8a\nINFO [alembic.runtime.migration] Running upgrade 0e66c5227a8a -> 45f8dd33480b\nINFO [alembic.runtime.migration] Running upgrade 45f8dd33480b -> 5abc0278ca73\nINFO [alembic.runtime.migration] Running upgrade 5abc0278ca73 -> d3435b514502\nINFO [alembic.runtime.migration] Running upgrade d3435b514502 -> 30107ab6a3ee\nINFO [alembic.runtime.migration] Running upgrade 30107ab6a3ee -> c415aab1c048\nINFO [alembic.runtime.migration] Running upgrade c415aab1c048 -> a963b38d82f4\nINFO [alembic.runtime.migration] Running upgrade kilo -> 30018084ec99\nINFO [alembic.runtime.migration] Running upgrade 30018084ec99 -> 4ffceebfada\nINFO [alembic.runtime.migration] Running upgrade 4ffceebfada -> 5498d17be016\nINFO [alembic.runtime.migration] Running upgrade 5498d17be016 -> 2a16083502f3\nINFO [alembic.runtime.migration] Running upgrade 2a16083502f3 -> 2e5352a0ad4d\nINFO [alembic.runtime.migration] Running upgrade 2e5352a0ad4d -> 11926bcfe72d\nINFO [alembic.runtime.migration] Running upgrade 11926bcfe72d -> 4af11ca47297\nINFO [alembic.runtime.migration] Running upgrade 4af11ca47297 -> 1b294093239c\nINFO [alembic.runtime.migration] Running upgrade 1b294093239c -> 8a6d8bdae39\nINFO [alembic.runtime.migration] Running upgrade 8a6d8bdae39 -> 2b4c2465d44b\nINFO [alembic.runtime.migration] Running upgrade 2b4c2465d44b -> e3278ee65050\nINFO [alembic.runtime.migration] Running upgrade e3278ee65050 -> c6c112992c9\nINFO [alembic.runtime.migration] Running upgrade c6c112992c9 -> 5ffceebfada\nINFO [alembic.runtime.migration] Running upgrade 5ffceebfada -> 4ffceebfcdc\nINFO [alembic.runtime.migration] Running upgrade 4ffceebfcdc -> 7bbb25278f53\nINFO [alembic.runtime.migration] Running upgrade 7bbb25278f53 -> 89ab9a816d70\nINFO [alembic.runtime.migration] Running upgrade 89ab9a816d70 -> c879c5e1ee90\nINFO [alembic.runtime.migration] Running upgrade c879c5e1ee90 -> 8fd3918ef6f4\nINFO [alembic.runtime.migration] Running upgrade 8fd3918ef6f4 -> 4bcd4df1f426\nINFO [alembic.runtime.migration] Running upgrade 4bcd4df1f426 -> b67e765a3524\nINFO [alembic.runtime.migration] Running upgrade a963b38d82f4 -> 3d0e74aa7d37\nINFO [alembic.runtime.migration] Running upgrade 3d0e74aa7d37 -> 030a959ceafa\nINFO [alembic.runtime.migration] Running upgrade 030a959ceafa -> a5648cfeeadf\nINFO [alembic.runtime.migration] Running upgrade a5648cfeeadf -> 0f5bef0f87d4\nINFO [alembic.runtime.migration] Running upgrade 0f5bef0f87d4 -> 67daae611b6e\nINFO [alembic.runtime.migration] Running upgrade b67e765a3524 -> a84ccf28f06a\nINFO [alembic.runtime.migration] Running upgrade a84ccf28f06a -> 7d9d8eeec6ad\nINFO [alembic.runtime.migration] Running upgrade 67daae611b6e -> 6b461a21bcfc\nINFO [alembic.runtime.migration] Running upgrade 6b461a21bcfc -> 5cd92597d11d\nINFO [alembic.runtime.migration] Running upgrade 5cd92597d11d -> 929c968efe70\nINFO [alembic.runtime.migration] Running upgrade 929c968efe70 -> a9c43481023c\nINFO [alembic.runtime.migration] Running upgrade a9c43481023c -> 804a3c76314c\nINFO [alembic.runtime.migration] Running upgrade 804a3c76314c -> 2b42d90729da\nINFO [alembic.runtime.migration] Running upgrade 2b42d90729da -> 62c781cb6192\nINFO [alembic.runtime.migration] Running upgrade 62c781cb6192 -> c8c222d42aa9\nINFO [alembic.runtime.migration] Running upgrade c8c222d42aa9 -> 349b6fd605a6\nINFO [alembic.runtime.migration] Running upgrade 349b6fd605a6 -> 7d32f979895f\nINFO [alembic.runtime.migration] Running upgrade 7d32f979895f -> 594422d373ee\nINFO [alembic.runtime.migration] Running upgrade 594422d373ee -> 61663558142c\nINFO [alembic.runtime.migration] Running upgrade 61663558142c -> 867d39095bf4, port forwarding\nINFO [alembic.runtime.migration] Running upgrade 867d39095bf4 -> d72db3e25539, modify uniq port forwarding\nINFO [alembic.runtime.migration] Running upgrade d72db3e25539 -> cada2437bf41\nINFO [alembic.runtime.migration] Running upgrade cada2437bf41 -> 195176fb410d, router gateway IP QoS\nINFO [alembic.runtime.migration] Running upgrade 195176fb410d -> fb0167bd9639\nINFO [alembic.runtime.migration] Running upgrade fb0167bd9639 -> 0ff9e3881597\nINFO [alembic.runtime.migration] Running upgrade 0ff9e3881597 -> 9bfad3f1e780\nINFO [alembic.runtime.migration] Running upgrade 9bfad3f1e780 -> 63fd95af7dcd\nINFO [alembic.runtime.migration] Running upgrade 63fd95af7dcd -> c613d0b82681\nINFO [alembic.runtime.migration] Running upgrade c613d0b82681 -> c3e9d13c4367\nINFO [alembic.runtime.migration] Running upgrade c3e9d13c4367 -> 86274d77933e\nINFO [alembic.runtime.migration] Running upgrade 86274d77933e -> f4b9654dd40c\nINFO [alembic.runtime.migration] Running upgrade f4b9654dd40c -> a010322604bc\nINFO [alembic.runtime.migration] Running upgrade a010322604bc -> 263d454a9655\nINFO [alembic.runtime.migration] Running upgrade 263d454a9655 -> Ibac91d24da2\nINFO [alembic.runtime.migration] Running upgrade Ibac91d24da2 -> 2217c4222de6\nINFO [alembic.runtime.migration] Running upgrade 2217c4222de6 -> 18a7e90ae768\nINFO [alembic.runtime.migration] Running upgrade 18a7e90ae768 -> e4e236b0e1ff\nINFO [alembic.runtime.migration] Running upgrade e4e236b0e1ff -> e88badaa9591\nINFO [alembic.runtime.migration] Running upgrade e88badaa9591 -> d8bdf05313f4\nINFO [alembic.runtime.migration] Running upgrade d8bdf05313f4 -> dfe425060830\nINFO [alembic.runtime.migration] Running upgrade dfe425060830 -> fd6107509ccd\nINFO [alembic.runtime.migration] Running upgrade fd6107509ccd -> 1ea5dab0897a\nINFO [alembic.runtime.migration] Running upgrade 1ea5dab0897a -> 49d8622c5221\nINFO [alembic.runtime.migration] Running upgrade 49d8622c5221 -> I38991de2b4\nINFO [alembic.runtime.migration] Running upgrade I38991de2b4 -> 532aa95457e2\nINFO [alembic.runtime.migration] Running upgrade 532aa95457e2 -> f010820fc498\nINFO [alembic.runtime.migration] Running upgrade f010820fc498 -> a964d94b4677\nINFO [alembic.runtime.migration] Running upgrade a964d94b4677 -> 26d1e9f5c766\nINFO [alembic.runtime.migration] Running upgrade 26d1e9f5c766 -> 1e0744e4ffea\nINFO [alembic.runtime.migration] Running upgrade 1e0744e4ffea -> 6135a7bd4425\nINFO [alembic.runtime.migration] Running upgrade 6135a7bd4425 -> 8df53b0d2c0e\nINFO [alembic.runtime.migration] Running upgrade 8df53b0d2c0e -> 1bb3393de75d, add qos policy rule Packet Rate Limit\nINFO [alembic.runtime.migration] Running upgrade 1bb3393de75d -> c181bb1d89e4\nINFO [alembic.runtime.migration] Running upgrade c181bb1d89e4 -> ba859d649675\nINFO [alembic.runtime.migration] Running upgrade ba859d649675 -> e981acd076d3\nINFO [alembic.runtime.migration] Running upgrade e981acd076d3 -> 76df7844a8c6, add Local IP tables\nINFO [alembic.runtime.migration] Running upgrade 76df7844a8c6 -> 1ffef8d6f371, migrate RBAC registers from \"target_tenant\" to \"target_project\"\nINFO [alembic.runtime.migration] Running upgrade 1ffef8d6f371 -> 8160f7a9cebb, drop portbindingports table\nINFO [alembic.runtime.migration] Running upgrade 8160f7a9cebb -> cd9ef14ccf87\nINFO [alembic.runtime.migration] Running upgrade cd9ef14ccf87 -> 34cf8b009713\nINFO [alembic.runtime.migration] Running upgrade 34cf8b009713 -> I43e0b669096\nINFO [alembic.runtime.migration] Running upgrade I43e0b669096 -> 4e6e655746f6\nINFO [alembic.runtime.migration] Running upgrade 4e6e655746f6 -> 659cbedf30a1\nINFO [alembic.runtime.migration] Running upgrade 659cbedf30a1 -> 21ff98fabab1\nINFO [alembic.runtime.migration] Running upgrade 21ff98fabab1 -> 5881373af7f5\nINFO [alembic.runtime.migration] Running upgrade 5881373af7f5 -> fc153938cdc1\nINFO [alembic.runtime.migration] Running upgrade fc153938cdc1 -> 93f394357a27\nINFO [alembic.runtime.migration] Running upgrade 93f394357a27 -> 6f1145bff34c\nINFO [alembic.runtime.migration] Running upgrade 6f1145bff34c -> 682c319773d7\nINFO [alembic.runtime.migration] Running upgrade 682c319773d7 -> 0aefee21cd87\nINFO [alembic.runtime.migration] Running upgrade 0aefee21cd87 -> b1199a3adbef\nINFO [alembic.runtime.migration] Running upgrade b1199a3adbef -> c33da356b165\nINFO [alembic.runtime.migration] Running upgrade c33da356b165 -> 89c58a70ceba\nINFO [alembic.runtime.migration] Running upgrade 89c58a70ceba -> 054e34dbe6b4\nINFO [alembic.runtime.migration] Running upgrade 054e34dbe6b4 -> 0e6eff810791\nINFO [alembic.runtime.migration] Running upgrade 0e6eff810791 -> 175fa80908e1\nINFO [alembic.runtime.migration] Running upgrade 175fa80908e1 -> 5bcb7b31ec7d\nINFO [alembic.runtime.migration] Running upgrade 5bcb7b31ec7d -> ad80a9f07c5c\nINFO [alembic.runtime.migration] Running upgrade 7d9d8eeec6ad -> a8b517cff8ab\nINFO [alembic.runtime.migration] Running upgrade a8b517cff8ab -> 3b935b28e7a0\nINFO [alembic.runtime.migration] Running upgrade 3b935b28e7a0 -> b12a3ef66e62\nINFO [alembic.runtime.migration] Running upgrade b12a3ef66e62 -> 97c25b0d2353\nINFO [alembic.runtime.migration] Running upgrade 97c25b0d2353 -> 2e0d7a8a1586\nINFO [alembic.runtime.migration] Running upgrade 2e0d7a8a1586 -> 5c85685d616d\n+++ [[ -n 0 ]]\n+++ for service in ${NEUTRON_BOOTSTRAP_SERVICES}\n+++ neutron-db-manage --subproject vmware-nsx upgrade head\nargument --subproject: Invalid String(choices=['networking-sfc', 'neutron-dynamic-routing', 'neutron-fwaas', 'neutron-vpnaas', 'neutron', 'tap-as-a-service']) value: vmware-nsx\n", "stderr_lines": ["+ sudo -E kolla_set_configs", "2026-01-06 12:53:23.464 INFO Loading config file at /var/lib/kolla/config_files/config.json", "2026-01-06 12:53:23.465 INFO Validating config file", "2026-01-06 12:53:23.465 INFO Kolla config strategy set to: COPY_ALWAYS", "2026-01-06 12:53:23.476 INFO Copying service configuration files", "2026-01-06 12:53:23.476 INFO Copying /var/lib/kolla/config_files/neutron.conf to /etc/neutron/neutron.conf", "2026-01-06 12:53:23.489 INFO Setting permission for /etc/neutron/neutron.conf", "2026-01-06 12:53:23.501 INFO Copying /var/lib/kolla/config_files/nsx.ini to /etc/neutron/plugins/vmware/nsx.ini", "2026-01-06 12:53:23.501 INFO Setting permission for /etc/neutron/plugins/vmware/nsx.ini", "2026-01-06 12:53:23.502 INFO Copying /var/lib/kolla/config_files/ml2_conf.ini to /etc/neutron/plugins/ml2/ml2_conf.ini", "2026-01-06 12:53:23.512 INFO Setting permission for /etc/neutron/plugins/ml2/ml2_conf.ini", "2026-01-06 12:53:23.523 INFO Copying /var/lib/kolla/config_files/id_rsa to /var/lib/neutron/.ssh/id_rsa", "2026-01-06 12:53:23.524 INFO Setting permission for /var/lib/neutron/.ssh/id_rsa", "2026-01-06 12:53:23.524 INFO Writing out command to execute", "++ cat /run_command", "+ CMD='neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/vmware/nsx.ini '", "+ ARGS=", "+ sudo kolla_copy_cacerts", "+ sudo kolla_install_projects", "+ [[ ! -n '' ]]", "+ . kolla_extend_start", "++ [[ ! -d /var/log/kolla/neutron ]]", "++ mkdir -p /var/log/kolla/neutron", "+++ stat -c %a /var/log/kolla/neutron", "++ [[ 2755 != \\7\\5\\5 ]]", "++ chmod 755 /var/log/kolla/neutron", "++ [[ ! ubuntu =~ centos|rocky ]]", "++ /usr/bin/update-alternatives --display iptables", "++ KOLLA_LEGACY_IPTABLES=false", "++ [[ false == \\t\\r\\u\\e ]]", "++ sudo /usr/bin/update-alternatives --auto iptables", "++ sudo /usr/bin/update-alternatives --auto ip6tables", "++ . /usr/local/bin/kolla_neutron_extend_start", "+++ [[ -n 0 ]]", "+++ neutron-db-manage --subproject neutron upgrade head", "INFO [alembic.runtime.migration] Context impl MySQLImpl.", "INFO [alembic.runtime.migration] Will assume non-transactional DDL.", "INFO [alembic.runtime.migration] Context impl MySQLImpl.", "INFO [alembic.runtime.migration] Will assume non-transactional DDL.", "INFO [alembic.runtime.migration] Running upgrade -> kilo", "INFO [alembic.runtime.migration] Running upgrade kilo -> 354db87e3225", "INFO [alembic.runtime.migration] Running upgrade 354db87e3225 -> 599c6a226151", "INFO [alembic.runtime.migration] Running upgrade 599c6a226151 -> 52c5312f6baf", "INFO [alembic.runtime.migration] Running upgrade 52c5312f6baf -> 313373c0ffee", "INFO [alembic.runtime.migration] Running upgrade 313373c0ffee -> 8675309a5c4f", "INFO [alembic.runtime.migration] Running upgrade 8675309a5c4f -> 45f955889773", "INFO [alembic.runtime.migration] Running upgrade 45f955889773 -> 26c371498592", "INFO [alembic.runtime.migration] Running upgrade 26c371498592 -> 1c844d1677f7", "INFO [alembic.runtime.migration] Running upgrade 1c844d1677f7 -> 1b4c6e320f79", "INFO [alembic.runtime.migration] Running upgrade 1b4c6e320f79 -> 48153cb5f051", "INFO [alembic.runtime.migration] Running upgrade 48153cb5f051 -> 9859ac9c136", "INFO [alembic.runtime.migration] Running upgrade 9859ac9c136 -> 34af2b5c5a59", "INFO [alembic.runtime.migration] Running upgrade 34af2b5c5a59 -> 59cb5b6cf4d", "INFO [alembic.runtime.migration] Running upgrade 59cb5b6cf4d -> 13cfb89f881a", "INFO [alembic.runtime.migration] Running upgrade 13cfb89f881a -> 32e5974ada25", "INFO [alembic.runtime.migration] Running upgrade 32e5974ada25 -> ec7fcfbf72ee", "INFO [alembic.runtime.migration] Running upgrade ec7fcfbf72ee -> dce3ec7a25c9", "INFO [alembic.runtime.migration] Running upgrade dce3ec7a25c9 -> c3a73f615e4", "INFO [alembic.runtime.migration] Running upgrade c3a73f615e4 -> 659bf3d90664", "INFO [alembic.runtime.migration] Running upgrade 659bf3d90664 -> 1df244e556f5", "INFO [alembic.runtime.migration] Running upgrade 1df244e556f5 -> 19f26505c74f", "INFO [alembic.runtime.migration] Running upgrade 19f26505c74f -> 15be73214821", "INFO [alembic.runtime.migration] Running upgrade 15be73214821 -> b4caf27aae4", "INFO [alembic.runtime.migration] Running upgrade b4caf27aae4 -> 15e43b934f81", "INFO [alembic.runtime.migration] Running upgrade 15e43b934f81 -> 31ed664953e6", "INFO [alembic.runtime.migration] Running upgrade 31ed664953e6 -> 2f9e956e7532", "INFO [alembic.runtime.migration] Running upgrade 2f9e956e7532 -> 3894bccad37f", "INFO [alembic.runtime.migration] Running upgrade 3894bccad37f -> 0e66c5227a8a", "INFO [alembic.runtime.migration] Running upgrade 0e66c5227a8a -> 45f8dd33480b", "INFO [alembic.runtime.migration] Running upgrade 45f8dd33480b -> 5abc0278ca73", "INFO [alembic.runtime.migration] Running upgrade 5abc0278ca73 -> d3435b514502", "INFO [alembic.runtime.migration] Running upgrade d3435b514502 -> 30107ab6a3ee", "INFO [alembic.runtime.migration] Running upgrade 30107ab6a3ee -> c415aab1c048", "INFO [alembic.runtime.migration] Running upgrade c415aab1c048 -> a963b38d82f4", "INFO [alembic.runtime.migration] Running upgrade kilo -> 30018084ec99", "INFO [alembic.runtime.migration] Running upgrade 30018084ec99 -> 4ffceebfada", "INFO [alembic.runtime.migration] Running upgrade 4ffceebfada -> 5498d17be016", "INFO [alembic.runtime.migration] Running upgrade 5498d17be016 -> 2a16083502f3", "INFO [alembic.runtime.migration] Running upgrade 2a16083502f3 -> 2e5352a0ad4d", "INFO [alembic.runtime.migration] Running upgrade 2e5352a0ad4d -> 11926bcfe72d", "INFO [alembic.runtime.migration] Running upgrade 11926bcfe72d -> 4af11ca47297", "INFO [alembic.runtime.migration] Running upgrade 4af11ca47297 -> 1b294093239c", "INFO [alembic.runtime.migration] Running upgrade 1b294093239c -> 8a6d8bdae39", "INFO [alembic.runtime.migration] Running upgrade 8a6d8bdae39 -> 2b4c2465d44b", "INFO [alembic.runtime.migration] Running upgrade 2b4c2465d44b -> e3278ee65050", "INFO [alembic.runtime.migration] Running upgrade e3278ee65050 -> c6c112992c9", "INFO [alembic.runtime.migration] Running upgrade c6c112992c9 -> 5ffceebfada", "INFO [alembic.runtime.migration] Running upgrade 5ffceebfada -> 4ffceebfcdc", "INFO [alembic.runtime.migration] Running upgrade 4ffceebfcdc -> 7bbb25278f53", "INFO [alembic.runtime.migration] Running upgrade 7bbb25278f53 -> 89ab9a816d70", "INFO [alembic.runtime.migration] Running upgrade 89ab9a816d70 -> c879c5e1ee90", "INFO [alembic.runtime.migration] Running upgrade c879c5e1ee90 -> 8fd3918ef6f4", "INFO [alembic.runtime.migration] Running upgrade 8fd3918ef6f4 -> 4bcd4df1f426", "INFO [alembic.runtime.migration] Running upgrade 4bcd4df1f426 -> b67e765a3524", "INFO [alembic.runtime.migration] Running upgrade a963b38d82f4 -> 3d0e74aa7d37", "INFO [alembic.runtime.migration] Running upgrade 3d0e74aa7d37 -> 030a959ceafa", "INFO [alembic.runtime.migration] Running upgrade 030a959ceafa -> a5648cfeeadf", "INFO [alembic.runtime.migration] Running upgrade a5648cfeeadf -> 0f5bef0f87d4", "INFO [alembic.runtime.migration] Running upgrade 0f5bef0f87d4 -> 67daae611b6e", "INFO [alembic.runtime.migration] Running upgrade b67e765a3524 -> a84ccf28f06a", "INFO [alembic.runtime.migration] Running upgrade a84ccf28f06a -> 7d9d8eeec6ad", "INFO [alembic.runtime.migration] Running upgrade 67daae611b6e -> 6b461a21bcfc", "INFO [alembic.runtime.migration] Running upgrade 6b461a21bcfc -> 5cd92597d11d", "INFO [alembic.runtime.migration] Running upgrade 5cd92597d11d -> 929c968efe70", "INFO [alembic.runtime.migration] Running upgrade 929c968efe70 -> a9c43481023c", "INFO [alembic.runtime.migration] Running upgrade a9c43481023c -> 804a3c76314c", "INFO [alembic.runtime.migration] Running upgrade 804a3c76314c -> 2b42d90729da", "INFO [alembic.runtime.migration] Running upgrade 2b42d90729da -> 62c781cb6192", "INFO [alembic.runtime.migration] Running upgrade 62c781cb6192 -> c8c222d42aa9", "INFO [alembic.runtime.migration] Running upgrade c8c222d42aa9 -> 349b6fd605a6", "INFO [alembic.runtime.migration] Running upgrade 349b6fd605a6 -> 7d32f979895f", "INFO [alembic.runtime.migration] Running upgrade 7d32f979895f -> 594422d373ee", "INFO [alembic.runtime.migration] Running upgrade 594422d373ee -> 61663558142c", "INFO [alembic.runtime.migration] Running upgrade 61663558142c -> 867d39095bf4, port forwarding", "INFO [alembic.runtime.migration] Running upgrade 867d39095bf4 -> d72db3e25539, modify uniq port forwarding", "INFO [alembic.runtime.migration] Running upgrade d72db3e25539 -> cada2437bf41", "INFO [alembic.runtime.migration] Running upgrade cada2437bf41 -> 195176fb410d, router gateway IP QoS", "INFO [alembic.runtime.migration] Running upgrade 195176fb410d -> fb0167bd9639", "INFO [alembic.runtime.migration] Running upgrade fb0167bd9639 -> 0ff9e3881597", "INFO [alembic.runtime.migration] Running upgrade 0ff9e3881597 -> 9bfad3f1e780", "INFO [alembic.runtime.migration] Running upgrade 9bfad3f1e780 -> 63fd95af7dcd", "INFO [alembic.runtime.migration] Running upgrade 63fd95af7dcd -> c613d0b82681", "INFO [alembic.runtime.migration] Running upgrade c613d0b82681 -> c3e9d13c4367", "INFO [alembic.runtime.migration] Running upgrade c3e9d13c4367 -> 86274d77933e", "INFO [alembic.runtime.migration] Running upgrade 86274d77933e -> f4b9654dd40c", "INFO [alembic.runtime.migration] Running upgrade f4b9654dd40c -> a010322604bc", "INFO [alembic.runtime.migration] Running upgrade a010322604bc -> 263d454a9655", "INFO [alembic.runtime.migration] Running upgrade 263d454a9655 -> Ibac91d24da2", "INFO [alembic.runtime.migration] Running upgrade Ibac91d24da2 -> 2217c4222de6", "INFO [alembic.runtime.migration] Running upgrade 2217c4222de6 -> 18a7e90ae768", "INFO [alembic.runtime.migration] Running upgrade 18a7e90ae768 -> e4e236b0e1ff", "INFO [alembic.runtime.migration] Running upgrade e4e236b0e1ff -> e88badaa9591", "INFO [alembic.runtime.migration] Running upgrade e88badaa9591 -> d8bdf05313f4", "INFO [alembic.runtime.migration] Running upgrade d8bdf05313f4 -> dfe425060830", "INFO [alembic.runtime.migration] Running upgrade dfe425060830 -> fd6107509ccd", "INFO [alembic.runtime.migration] Running upgrade fd6107509ccd -> 1ea5dab0897a", "INFO [alembic.runtime.migration] Running upgrade 1ea5dab0897a -> 49d8622c5221", "INFO [alembic.runtime.migration] Running upgrade 49d8622c5221 -> I38991de2b4", "INFO [alembic.runtime.migration] Running upgrade I38991de2b4 -> 532aa95457e2", "INFO [alembic.runtime.migration] Running upgrade 532aa95457e2 -> f010820fc498", "INFO [alembic.runtime.migration] Running upgrade f010820fc498 -> a964d94b4677", "INFO [alembic.runtime.migration] Running upgrade a964d94b4677 -> 26d1e9f5c766", "INFO [alembic.runtime.migration] Running upgrade 26d1e9f5c766 -> 1e0744e4ffea", "INFO [alembic.runtime.migration] Running upgrade 1e0744e4ffea -> 6135a7bd4425", "INFO [alembic.runtime.migration] Running upgrade 6135a7bd4425 -> 8df53b0d2c0e", "INFO [alembic.runtime.migration] Running upgrade 8df53b0d2c0e -> 1bb3393de75d, add qos policy rule Packet Rate Limit", "INFO [alembic.runtime.migration] Running upgrade 1bb3393de75d -> c181bb1d89e4", "INFO [alembic.runtime.migration] Running upgrade c181bb1d89e4 -> ba859d649675", "INFO [alembic.runtime.migration] Running upgrade ba859d649675 -> e981acd076d3", "INFO [alembic.runtime.migration] Running upgrade e981acd076d3 -> 76df7844a8c6, add Local IP tables", "INFO [alembic.runtime.migration] Running upgrade 76df7844a8c6 -> 1ffef8d6f371, migrate RBAC registers from \"target_tenant\" to \"target_project\"", "INFO [alembic.runtime.migration] Running upgrade 1ffef8d6f371 -> 8160f7a9cebb, drop portbindingports table", "INFO [alembic.runtime.migration] Running upgrade 8160f7a9cebb -> cd9ef14ccf87", "INFO [alembic.runtime.migration] Running upgrade cd9ef14ccf87 -> 34cf8b009713", "INFO [alembic.runtime.migration] Running upgrade 34cf8b009713 -> I43e0b669096", "INFO [alembic.runtime.migration] Running upgrade I43e0b669096 -> 4e6e655746f6", "INFO [alembic.runtime.migration] Running upgrade 4e6e655746f6 -> 659cbedf30a1", "INFO [alembic.runtime.migration] Running upgrade 659cbedf30a1 -> 21ff98fabab1", "INFO [alembic.runtime.migration] Running upgrade 21ff98fabab1 -> 5881373af7f5", "INFO [alembic.runtime.migration] Running upgrade 5881373af7f5 -> fc153938cdc1", "INFO [alembic.runtime.migration] Running upgrade fc153938cdc1 -> 93f394357a27", "INFO [alembic.runtime.migration] Running upgrade 93f394357a27 -> 6f1145bff34c", "INFO [alembic.runtime.migration] Running upgrade 6f1145bff34c -> 682c319773d7", "INFO [alembic.runtime.migration] Running upgrade 682c319773d7 -> 0aefee21cd87", "INFO [alembic.runtime.migration] Running upgrade 0aefee21cd87 -> b1199a3adbef", "INFO [alembic.runtime.migration] Running upgrade b1199a3adbef -> c33da356b165", "INFO [alembic.runtime.migration] Running upgrade c33da356b165 -> 89c58a70ceba", "INFO [alembic.runtime.migration] Running upgrade 89c58a70ceba -> 054e34dbe6b4", "INFO [alembic.runtime.migration] Running upgrade 054e34dbe6b4 -> 0e6eff810791", "INFO [alembic.runtime.migration] Running upgrade 0e6eff810791 -> 175fa80908e1", "INFO [alembic.runtime.migration] Running upgrade 175fa80908e1 -> 5bcb7b31ec7d", "INFO [alembic.runtime.migration] Running upgrade 5bcb7b31ec7d -> ad80a9f07c5c", "INFO [alembic.runtime.migration] Running upgrade 7d9d8eeec6ad -> a8b517cff8ab", "INFO [alembic.runtime.migration] Running upgrade a8b517cff8ab -> 3b935b28e7a0", "INFO [alembic.runtime.migration] Running upgrade 3b935b28e7a0 -> b12a3ef66e62", "INFO [alembic.runtime.migration] Running upgrade b12a3ef66e62 -> 97c25b0d2353", "INFO [alembic.runtime.migration] Running upgrade 97c25b0d2353 -> 2e0d7a8a1586", "INFO [alembic.runtime.migration] Running upgrade 2e0d7a8a1586 -> 5c85685d616d", "+++ [[ -n 0 ]]", "+++ for service in ${NEUTRON_BOOTSTRAP_SERVICES}", "+++ neutron-db-manage --subproject vmware-nsx upgrade head", "argument --subproject: Invalid String(choices=['networking-sfc', 'neutron-dynamic-routing', 'neutron-fwaas', 'neutron-vpnaas', 'neutron', 'tap-as-a-service']) value: vmware-nsx"], "stdout": "iptables - auto mode\n link best version is /usr/sbin/iptables-nft\n link currently points to /usr/sbin/iptables-nft\n link iptables is /usr/sbin/iptables\n slave iptables-restore is /usr/sbin/iptables-restore\n slave iptables-save is /usr/sbin/iptables-save\n/usr/sbin/iptables-legacy - priority 10\n slave iptables-restore: /usr/sbin/iptables-legacy-restore\n slave iptables-save: /usr/sbin/iptables-legacy-save\n/usr/sbin/iptables-nft - priority 20\n slave iptables-restore: /usr/sbin/iptables-nft-restore\n slave iptables-save: /usr/sbin/iptables-nft-save\nRunning upgrade for neutron ...\nOK\n", "stdout_lines": ["iptables - auto mode", " link best version is /usr/sbin/iptables-nft", " link currently points to /usr/sbin/iptables-nft", " link iptables is /usr/sbin/iptables", " slave iptables-restore is /usr/sbin/iptables-restore", " slave iptables-save is /usr/sbin/iptables-save", "/usr/sbin/iptables-legacy - priority 10", " slave iptables-restore: /usr/sbin/iptables-legacy-restore", " slave iptables-save: /usr/sbin/iptables-legacy-save", "/usr/sbin/iptables-nft - priority 20", " slave iptables-restore: /usr/sbin/iptables-nft-restore", " slave iptables-save: /usr/sbin/iptables-nft-save", "Running upgrade for neutron ...", "OK"]} whats the problem ?

2 3

[tc][all] No weekly IRC meeting on January 6 2026
by Goutham Pacha Ravi 06 Jan '26

06 Jan '26

Hello Stackers, A number of TC members are still out on PTO or observing public holidays in their respective countries, so we’ll be skipping the weekly IRC meeting this week, and the next meeting will be on January 13th 2026 at 1700 UTC in OFTC’s #openstack-tc channel. Thank you, Goutham Pacha Ravi (gouthamr) Openstack TC Chair

1 0

[openstacksdk][keystoneauth1][nova] Memory leak in nova-scheduler, possibly all services?
by Clippinger, Sam 05 Jan '26

05 Jan '26

Hello everyone, We've been seeing steady memory leaks in some of our OpenStack API services. I think I've found the cause but I'd appreciate some feedback. When a service creates a client object (e.g. placementclient) to use another service's API, the call goes through Proxy.request() in the openstacksdk module. That function calls _report_stats()[1] for every request which calls three other functions for statsd, prometheus and influxdb. The prometheus function records the number of requests and response times in a dict using the full URL of the request.[2] This data is recorded unless the _prometheus_counter or _prometheus_histogram member variables are None.[2] Those member variables are set by config.CloudRegion.get_prometheus_counter()[3] and config.CloudRegion.get_prometheus_histogram()[4], which do not check any config settings. It looks like the only way to prevent this behavior is to uninstall the prometheus_client module.[5] Unfortunately prometheus_client is required by oslo.metrics[6] which is required by oslo.messaging[7]. This is causing a lot of extra memory usage in our production environments. It's worst in our build farm where we create over 500K VMs per week. nova-scheduler queries a lot of unique URLs because the Placement API uses the VM's UUID to get allocations[8] and each URL gets its own counter and histogram. Even though we run 6 copies of nova-scheduler (currently Caracal) after a few weeks each copy will reach 8 GB and the worker processes will start getting OOMkilled, putting the new VM in error state. The nova-scheduler parent process keeps running and restarts the workers, which just keep getting killed and VMs can't be reliably scheduled until nova-scheduler is restarted. This feels like two problems to me. We don't need Prometheus metrics from the OpenStack services but there's no way to turn them off. The Session class in keystoneauth1 supports a collect-timing option[9] (default False) but it's only used in one or two places that I can find. The CloudRegion methods get their config from the Adapter[10] class in keystoneauth1, which does not support collect-timing. Should it? Even if we did use these Prometheus metrics, it seems like they shouldn't be allowed to grow unbounded forever. Should the Proxy._report_stats_prometheus() function[2] limit the metrics by age or quantity? Any thoughts? I've patched our services in-house to turn off these metrics (I'll have results in a week). In the meantime, are we doing something wrong or is every OpenStack service always collecting these metrics constantly? -- Sam Clippinger 1: https://github.com/openstack/openstacksdk/blob/stable/2025.2/openstack/prox… 2: https://github.com/openstack/openstacksdk/blob/stable/2025.2/openstack/prox… 3: https://github.com/openstack/openstacksdk/blob/stable/2025.2/openstack/conf… 4: https://github.com/openstack/openstacksdk/blob/stable/2025.2/openstack/conf… 5: https://github.com/openstack/openstacksdk/blob/stable/2025.2/openstack/conf… 6: https://github.com/openstack/oslo.metrics/blob/stable/2025.2/requirements.t… 7: https://github.com/openstack/oslo.messaging/blob/stable/2025.2/requirements… 8: https://docs.openstack.org/api-ref/placement/#list-allocations 9: https://github.com/openstack/keystoneauth/blob/stable/2025.2/keystoneauth1/… 10: https://github.com/openstack/keystoneauth/blob/stable/2025.2/keystoneauth1/… ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient(s) and contain information that may be Garmin confidential and/or Garmin legally privileged. If you have received this email in error, please notify the sender by reply email and delete the message. Any disclosure, copying, distribution or use of this communication (including attachments) by someone other than the intended recipient is prohibited. Thank you.

4 4