July 2019 - openstack-discuss - lists.openstack.org

[image-encryption][nova][cinder][glance][barbican]Image Encryption popup team meeting
by Josephine Seifert 10 Jul '19

10 Jul '19

Hello, starting with next week, there will be a weekly meeting on *Monday, 13 UTC *[1]**for the Image Encryption popup team. Please feel free to join us :) Greetings Josephine [1] http://eavesdrop.openstack.org/#Image_Encryption_Popup-Team_Meeting

1 0

[watcher] nova cdm builder performance optimizations - summary
by Matt Riedemann 10 Jul '19

10 Jul '19

I wanted to summarize a series of changes which have improved the performance of the NovaClusterDataModel builder for audits across single and multiple cells (in the CERN case) by a factor of 20-30%. There were initially three changes involved (in order): 1. https://review.opendev.org/#/c/659688/ - Optimize NovaClusterDataModelCollector.add_instance_node Reports on that patch alone said it fixed a regression introduced in Stein with scoped audits: "I checked this patch on the my test environment on the stable/stein branch. I have more than 1000 virtual servers (some real, some dummy). Previously, in the stable/rocky branch, the time to build a cluster was about 15-20 minutes, in the Stein branch there was a regression and the time increased to 90 minutes. After this patch, the build time is only 2 minutes." That change was backported to stable/stein. 2. - https://review.opendev.org/#/c/661121/ - Optimize hypervisor API calls (which requires https://review.opendev.org/#/c/659886/) As noted that change requires a patch to python-novaclient if you are looking to backport the change. We can't backport that upstream because of the python-novaclient dependency since it would require bumping the minimum required version of the library on a stable branch which is against stable branch policy (minimum version of library dependencies are more or less frozen on stable branches). That change also requires configuring watcher with: [nova_client] api_version = 2.53 # or greater; train now requires at least 2.56 3. - https://review.opendev.org/#/c/662089/ - Optimize NovaHelper.get_compute_node_by_hostname This optimizes code used to build/update the nova CDM during notification processing and also fixes a bug about looking up the compute service properly. After those three changes were merged, Corne Lukken (Dantali0n) started doing scale and performance testing with and without the changes in a CERN 5-cell test cluster. Corne identified a regression for which Canwei Li determined the root cause and chenker fixed: 4. https://review.opendev.org/#/c/668100/ - Reduce the query time of the instances when call get_instance_list() With that fix applied Corne reported the overall improvement of 20-30% when building the nova CDM during an audit in various scenarios. The actual performance numbers will be sent later as part of a thesis Corne is working on. I want to thank Dantali0n, licanwei and chenker for all of their help with this series of improvements. -- Thanks, Matt

2 1

[nova][ironic] Lock-related performance issue with update_resources periodic job
by Jason Anderson 09 Jul '19

09 Jul '19

Hey OpenStackers, I work on a cloud that allows users to reserve and provision bare metal instances with Ironic. We recently performed a long-overdue upgrade of our core components, all the way from Ocata up through Rocky. During this, we noticed that instance build requests were taking 4-5x (!!) as long as before. We have two deployments, one with ~150 bare metal nodes, and another with ~300. These are each managed by one nova-compute process running the Ironic driver. After investigation, the root cause appeared to be contention between the update_resources periodic task and the instance claim step. There is one semaphore "compute_resources" that is used to control every access within the resource_tracker. In our case, what was happening was the update_resources job, which runs every minute by default, was constantly queuing up accesses to this semaphore, because each hypervisor is updated independently, in series. This meant that, for us, each Ironic node was being processed and was holding the semaphore during its update (which took about 2-5 seconds in practice.) Multiply this by 150 and our update task was running constantly. Because an instance claim also needs to access this semaphore, this led to instances getting stuck in the "Build" state, after scheduling, for tens of minutes on average. There seemed to be some probabilistic effect here, which I hypothesize is related to the locking mechanism not using a "fair" lock (first-come, first-served) by default. Our fix was to drastically increase the interval this task runs at--from every 1 minute to every 12 hours. We only provision bare metal, so my rationale was that the periodic full resource sync was less important and mostly helpful for fixing weird things where somehow Placement's state got out of sync with Nova's somehow. I'm wondering, after all this, if it makes sense to rethink this one-semaphore thing, and instead create a per-hypervisor semaphore when doing the resource syncing. I can't think of a reason why the entire set of hypervisors needs to be considered as a whole when doing this task, but I could very well be missing something. TL;DR: if you have one nova-compute process managing lots of Ironic hypervisors, consider tweaking the update_resources_interval to a higher value, especially if you're seeing instances stuck in the Build state for a while. Cheers, Jason Anderson Cloud Computing Software Developer Consortium for Advanced Science and Engineering, The University of Chicago Mathematics & Computer Science Division, Argonne National Laboratory

5 11

[scientific-sig] IRC today
by Stig Telfer 09 Jul '19

09 Jul '19

Hi All - We have a Scientific SIG IRC meeting coming up at 2100 UTC (about 15 minutes time) in channel #openstack-meeting. Everyone is welcome. Today’s agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_July_9th_2019 If you’d like to add anything to the agenda, come along to the meeting and bring it up. Cheers, Stig

1 0

[tripleo] [cisco] [bigswitch] [midonet] Composable Services for Neutron plugins
by Emilien Macchi 09 Jul '19

09 Jul '19

With that thread I'm hoping to reach out some of the folks who were involved in TripleO services for some of the Neutron plugins like Cisco, Bigswitch and Midonet. Some of them still use ExtraConfig interface which isn't super ideal. I'm willing to help convert them into composable services but I'll need help on the reviews and more importantly the actual testing. If you're in CC and can help, good please let me know. If you're not involved anymore, please give me a name if possible. If you were involved and know the plugin doesn't need maintenance, let me know as well, it'll save me time. I started with the Cisco UCSM: https://review.opendev.org/669931 -- Emilien Macchi

1 0

[cinder][stable-maint] July releases from the stable branches
by Brian Rosmaita 09 Jul '19

09 Jul '19

I've posted release patches for the stable branches: https://review.openstack.org/#/q/topic:cinderproject-july-2019 Notes on what is/isn't being released are on the etherpad: https://etherpad.openstack.org/p/cinder-releases-tracking I have a question about the semver for the cinder stable/stein release; it's noted on the patch: https://review.opendev.org/669771 cheers, brian

2 1

[Glance][PTL] Vacation time
by Erno Kuvaja 09 Jul '19

09 Jul '19

Hi all, I'll be away for couple of weeks. I'll be back Wed 24th of July. - Erno "jokke" Kuvaja

1 0

[docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos
by Mohammed Naser 09 Jul '19

09 Jul '19

Hi everyone, One of the issue that we recently ran into was the fact that there was some inconsistency about merging retirement of repositories inside governance without the code being fully removed. In order to avoid this, I've made a change to our governance repository which will enforce that no code exists in those retired repositories, however, this has surfaced that some repositories were retired with some stale files, some are smaller littler files, some are entire projects still. I have compiled a list for every team, with the repos that are not properly retired that have extra files (using this change which should eventually +1 once we fix it all: https://review.opendev.org/669549) [Documentation] openstack/api-site has extra files, please remove: .gitignore, .zuul.yaml, LICENSE, api-quick-start, api-ref, bindep.txt, common, doc-tools-check-languages.conf, firstapp, test-requirements.txt, tools, tox.ini, www [Documentation] openstack/faafo has extra files, please remove: .gitignore, CONTRIBUTING.rst, LICENSE, Vagrantfile, bin, contrib, doc, etc, faafo, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini [fuel] openstack/fuel-agent has extra files, please remove: .gitignore, LICENSE, MAINTAINERS, cloud-init-templates, contrib, debian, etc, fuel_agent, requirements.txt, run_tests.sh, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [fuel] openstack/fuel-astute has extra files, please remove: .gitignore, .rspec, .ruby-version, Gemfile, LICENSE, MAINTAINERS, Rakefile, astute.gemspec, astute.service, astute.sysconfig, bin, bindep.txt, debian, examples, lib, mcagents, run_tests.sh, spec, specs, tests [fuel] openstack/fuel-library has extra files, please remove: .gitignore, CHANGELOG, Gemfile, LICENSE, MAINTAINERS, Rakefile, debian, deployment, files, graphs, logs, specs, tests, utils [fuel] openstack/fuel-main has extra files, please remove: .gitignore, 00-debmirror.patch, LICENSE, MAINTAINERS, Makefile, config.mk, fuel-release, iso, mirror, packages, prepare-build-env.sh, report-changelog.sh, repos.mk, requirements-fuel-rpm.txt, requirements-rpm.txt, rules.mk, sandbox.mk, specs [fuel] openstack/fuel-menu has extra files, please remove: .gitignore, MAINTAINERS, MANIFEST.in, fuelmenu, run_tests.sh, setup.py, specs, test-requirements.txt, tox.ini [fuel] openstack/fuel-mirror has extra files, please remove: .gitignore, .mailmap, MAINTAINERS, perestroika, tox.ini [fuel] openstack/fuel-nailgun-agent has extra files, please remove: .gitignore, Gemfile, LICENSE, MAINTAINERS, Rakefile, agent, debian, nailgun-agent.cron, nailgun-agent.gemspec, run_tests.sh, specs [fuel] openstack/fuel-ostf has extra files, please remove: .gitignore, LICENSE, MAINTAINERS, MANIFEST.in, etc, fuel_health, fuel_plugin, ostf.service, pylintrc, requirements.txt, run_tests.sh, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [fuel] openstack/fuel-qa has extra files, please remove: .coveragerc, .gitignore, .pylintrc, .pylintrc_gerrit, MAINTAINERS, core, doc, fuel_tests, fuelweb_test, gates_tests, packages_tests, pytest.ini, run_system_test.py, run_tests.sh, system_test, tox.ini, utils [fuel] openstack/fuel-ui has extra files, please remove: .eslintignore, .eslintrc.yaml, .gitignore, LICENSE, MAINTAINERS, fixtures, gulp, gulpfile.js, karma.config.js, npm-shrinkwrap.json, package.json, run_real_plugin_tests.sh, run_real_plugin_tests_on_real_nailgun.sh, run_ui_func_tests.sh, specs, static, webpack.config.js [fuel] openstack/fuel-virtualbox has extra files, please remove: .gitignore, MAINTAINERS, actions, clean.sh, config.sh, contrib, drivers, dumpkeys.cache, functions, iso, launch.sh, launch_16GB.sh, launch_8GB.sh [fuel] openstack/fuel-web has extra files, please remove: .gitignore, LICENSE, MAINTAINERS, bin, build_docs.sh, debian, docs, nailgun, run_tests.sh, specs, systemd, tools, tox.ini [fuel] openstack/shotgun has extra files, please remove: .coveragerc, .gitignore, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, MAINTAINERS, MANIFEST.in, bin, etc, requirements.txt, setup.cfg, setup.py, shotgun, specs, test-requirements.txt, tox.ini [fuel] openstack/fuel-dev-tools has extra files, please remove: .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, MAINTAINERS, babel.cfg, contrib, doc, fuel_dev_tools, openstack-common.conf, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini, vagrant [fuel] openstack/fuel-devops has extra files, please remove: .coveragerc, .gitignore, .pylintrc, .pylintrc_gerrit, LICENSE, MAINTAINERS, bin, devops, doc, run_tests.sh, samples, setup.cfg, setup.py, test-requirements.txt, tox.ini [fuel] openstack/fuel-docs has extra files, please remove: .gitignore, Makefile, _images, _templates, common_conf.py, conf.py, devdocs, examples, glossary, index.rst, make.bat, plugindocs, requirements.txt, setup.cfg, setup.py, tox.ini, userdocs [fuel] openstack/fuel-nailgun-extension-cluster-upgrade has extra files, please remove: .coveragerc, .gitignore, AUTHORS, LICENSE, MANIFEST.in, bindep.txt, cluster_upgrade, conftest.py, nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [fuel] openstack/fuel-nailgun-extension-iac has extra files, please remove: .gitignore, LICENSE, MANIFEST.in, doc, fuel_external_git, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [fuel] openstack/fuel-nailgun-extension-converted-serializers has extra files, please remove: .coveragerc, .gitignore, LICENSE, MANIFEST.in, bindep.txt, conftest.py, converted_serializers, nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [fuel] openstack/fuel-octane has extra files, please remove: .coveragerc, .gitignore, .mailmap, Gemfile, Gemfile.lock, HACKING.rst, LICENSE, MAINTAINERS, MANIFEST.in, Rakefile, bindep.txt, deploy, deployment, docs, misc, octane, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tox.ini [fuel] openstack/fuel-upgrade has extra files, please remove: .gitignore [fuel] openstack/tuning-box has extra files, please remove: .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, MAINTAINERS, MANIFEST.in, TODO, alembic.ini, babel.cfg, bindep.txt, doc, examples, openstack-common.conf, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini, tuning_box [fuel] openstack/fuel-plugins has extra files, please remove: .gitignore, CHANGELOG.md, CONTRIBUTING.rst, HACKING.rst, LICENSE, MAINTAINERS, examples, fuel_plugin_builder, requirements.txt, run_tests.sh, setup.cfg, setup.py, test-requirements.txt, tox.ini [fuel] openstack/fuel-plugin-murano has extra files, please remove: .gitignore, LICENSE, components.yaml, deployment_scripts, deployment_tasks.yaml, docs, environment_config.yaml, functions.sh, metadata.yaml, node_roles.yaml, pre_build_hook, releasenotes, repositories, test-requirements.txt, tox.ini, volumes.yaml [fuel] openstack/fuel-plugin-murano-tests has extra files, please remove: .gitignore, murano_plugin_tests, openrc.default, requirements.txt, tox.ini, utils [fuel] openstack/fuel-specs has extra files, please remove: .gitignore, .testr.conf, LICENSE, doc, images, policy, requirements.txt, setup.cfg, setup.py, specs, tests, tools, tox.ini [fuel] openstack/fuel-stats has extra files, please remove: .gitignore, LICENSE, MAINTAINERS, MANIFEST.in, analytics, collector, migration, requirements.txt, setup.py, test-requirements.txt, tools, tox.ini [fuel] openstack/python-fuelclient has extra files, please remove: .gitignore, .testr.conf, MAINTAINERS, MANIFEST.in, fuelclient, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [Infrastructure] opendev/puppet-releasestatus has extra files, please remove: .gitignore [ironic] openstack/python-dracclient has extra files, please remove: .gitignore, CONTRIBUTING.rst, HACKING.rst, LICENSE, doc, dracclient, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini [neutron] openstack/networking-calico has extra files, please remove: .coveragerc, .gitignore, .mailmap, .testr.conf, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, RELEASING.md, babel.cfg, debian, devstack, doc, networking_calico, playbooks, requirements.txt, rpm, setup.cfg, setup.py, test-requirements.txt, tox.ini [neutron] openstack/networking-l2gw has extra files, please remove: .coveragerc, .gitignore, .testr.conf, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, bindep.txt, contrib, debian, devstack, doc, etc, lower-constraints.txt, networking_l2gw, openstack-common.conf, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [neutron] openstack/networking-l2gw-tempest-plugin has extra files, please remove: .gitignore, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, babel.cfg, contrib, doc, networking_l2gw_tempest_plugin, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini [neutron] openstack/networking-onos has extra files, please remove: .coveragerc, .gitignore, .mailmap, .pylintrc, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, PKG-INFO, TESTING.rst, babel.cfg, devstack, doc, etc, lower-constraints.txt, networking_onos, package, rally-jobs, releasenotes, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tools, tox.ini [neutron] openstack/neutron-vpnaas has extra files, please remove: .coveragerc, .gitignore, .mailmap, .pylintrc, .stestr.conf, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, TESTING.rst, babel.cfg, devstack, doc, etc, lower-constraints.txt, neutron_vpnaas, playbooks, rally-jobs, releasenotes, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tools, tox.ini [OpenStack Charms] openstack/charm-ceph has extra files, please remove: .gitignore [OpenStackAnsible] openstack/openstack-ansible-os_monasca has extra files, please remove: tests, tox.ini [solum] openstack/solum-infra-guestagent has extra files, please remove: .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, config-generator, doc, etc, requirements.txt, setup.cfg, setup.py, solum_guestagent, test-requirements.txt, tox.ini I'd like to kindly ask the affected teams to help out with this, or any member of our community is more than welcome to push a change to those repos and work with the appropriate teams to help land it. Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

8 11

neutron netns arp issue
by Satish Patel 09 Jul '19

09 Jul '19

Hello, I am deploying openstack-ansible with octavia and i can see neutron created network for lb-mgmt-net which also created dhcp namespace for that network which is in vlan27 so far everything good so for testing i have created vm and it didn't get IP address so i have started troubleshooting and i found my namespace sending arp request but not getting reply back. [root@ostack-infra-2-2 ~]# ip netns exec qdhcp-2b94d9df-dd49-45b5-a992-63fee27bfa77 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ns-5604eec1-20@if132: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:c2:b3:4d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.27.12.3/21 brd 172.27.15.255 scope global ns-5604eec1-20 valid_lft forever preferred_lft forever inet 169.254.169.254/16 brd 169.254.255.255 scope global ns-5604eec1-20 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fec2:b34d/64 scope link valid_lft forever preferred_lft forever This is my linuxbridge [root@ostack-infra-2-2 ~]# brctl show brq2b94d9df-dd bridge name bridge id STP enabled interfaces brq2b94d9df-dd 8000.16d25dbea2cc no br-vlan.27 tap5604eec1-20 on same controller node "ostack-infra-2-2" i have br-lbaas network which has same VLAN 27 subnet IP. now when i ping from dhcp-namespace to outside host on same vlan 27, i can see ARP going out and remote host replying back but my reply coming on br-lbaas interface. [root@ostack-infra-2-2 ~]# ip netns exec qdhcp-2b94d9df-dd49-45b5-a992-63fee27bfa77 ping 172.27.8.4 PING 172.27.8.4 (172.27.8.4) 56(84) bytes of data. >From 172.27.12.3 icmp_seq=1 Destination Host Unreachable >From 172.27.12.3 icmp_seq=2 Destination Host Unreachable on other terminal i am running tcpdump on br-lbaas and i am seeing remote host ARP reply coming on that interface but not going to br-vlan.27 which neutron created. [root@ostack-infra-2-2 network-scripts]# tcpdump -i br-lbaas -nn tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on br-lbaas, link-type EN10MB (Ethernet), capture size 262144 bytes 22:41:38.920858 ARP, Reply 172.27.8.4 is-at 32:7c:a1:91:79:7c, length 46 22:41:39.922167 ARP, Reply 172.27.8.4 is-at 32:7c:a1:91:79:7c, length 46 22:41:40.924052 ARP, Reply 172.27.8.4 is-at 32:7c:a1:91:79:7c, length 46 Do you think i can't create two same subnet bridge on same host?

1 0

[Neutron] NUMA aware VxLAN
by Guo, Ruijing 09 Jul '19

09 Jul '19

Hi, Existing neutron ML2 support one VxLAN for tenant network. In NUMA case, VM 0 can be created in node 0 and VM 1 can be created in node 1 and VxLAN is in node 0. VM1 need to cross node, which cause some performance downgrade. Does someone have this performance issue? Does Neutron community have plan to enhance it? Thanks, -Ruijing

3 2