June 2019 - openstack-discuss - lists.openstack.org

[Kolla] Stuck at bootstrap_gnocchi during deployment using Ubuntu binary on Rocky release.
by Eddie Yen 11 Jun '19

11 Jun '19

Hi Our env needs Gnocchi because Ceilometer, and we using Kolla to deploy Ceph as the env storage backend. But I found that it always stuck at bootstrap_gnocchi. The error log shows below when check with docker logs: 2019-06-11 10:59:17,707 [19] ERROR gnocchi.utils: Unable to initialize storage driver Traceback (most recent call last): File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 333, in call result = fn(*args, **kwargs) File "/usr/lib/python3/dist-packages/gnocchi/storage/__init__.py", line 102, in get_driver conf.storage) File "/usr/lib/python3/dist-packages/gnocchi/storage/ceph.py", line 52, in __init__ self.rados, self.ioctx = ceph.create_rados_connection(conf) File "/usr/lib/python3/dist-packages/gnocchi/common/ceph.py", line 51, in create_rados_connection raise ImportError("No module named 'rados' nor 'cradox'") ImportError: No module named 'rados' nor 'cradox' This error occurred not only the image from Docker Hub, but also build by Kolla-build in Ubuntu binary based. Strange is, no error occur if turn into use Ubuntu source based images. And I guess it only happen when enabled Ceph. Does anyone have idea about this? Many thanks, Eddie.

2 2

[tripleo] TripleO CI Summary: Sprint 31
by Rafael Folco 10 Jun '19

10 Jun '19

Greetings, The TripleO CI team has just completed Sprint 31 / Unified Sprint 10 (May 16 thru Jun 05). The following is a summary of completed work during this sprint cycle: - Created image and container build jobs for RDO on RHEL 7 in the internal instance of Software Factory. - Completed the bootstrapping of OSP 15 standalone job on RHEL8 running in the internal Software Factory. - Promotion status: green on all branches at most of the sprint. The planned work for the next sprint [1] are: - Complete RDO on RHEL7 work by having an independent pipeline running container and image build, standalone and ovb featureset001 jobs. This includes fixing ovb job and start consuming rhel containers from the standalone jobs. - Replicate RHEL7 jobs created in the last sprint for RHEL8 running in the internal Software Factory. Expected outcome is to have a preliminary job producing logs with successes or failures at the end of the sprint. - Create a design document for a staging environment to test changes in the promoter server. This will benefit CI team with less breakages in the promoter server and also prepare the grounds for the multi-arch builds. The Ruck and Rover for this sprint are Sorin Sbarnea (zbr) and Ronelle Landy (rlandy). Please direct questions or queries to them regarding CI status or issues in #tripleo, ideally to whomever has the ‘|ruck’ suffix on their nick. Ruck/rover notes are being tracked in etherpad [2]. Thanks, rfolco [1] https://tree.taiga.io/project/tripleo-ci-board/taskboard/unified-sprint-11 [2] https://etherpad.openstack.org/p/ruckroversprint11

1 0

[ops][neutron]After an upgrade to openstack queens, Neutron is unable to communicate properly with rabbitmq
by Jean-Philippe Méthot 10 Jun '19

10 Jun '19

Hi, We had a Pike openstack setup that we updated to Queens earlier this week. It’s a 30 compute nodes infrastructure with 2 controller nodes and 2 network nodes, using openvswitch for networking. Since we upgraded to queens, neutron-server on the controller nodes has been unable to contact the openvswitch-agents through rabbitmq. The rabbitmq is clustered on both controller nodes and has been giving us the following error when neutron-server connections fail : =ERROR REPORT==== 5-Jun-2019::18:50:08 === closing AMQP connection <0.23859.0> (10.30.0.11:53198 -> 10.30.0.11:5672 - neutron-server:1170:ccf11f31-2b3b-414e-ab19-5ee2cf5dd15d): missed heartbeats from client, timeout: 60s The neutron-server logs show this error: 2019-06-05 18:50:33.132 1169 ERROR oslo.messaging._drivers.impl_rabbit [req-17167988-c6f2-475e-8b6a-90b92777e03a - - - - -] [b7684919-c98b-402e-90c3-59a0b5eccd1f] AMQP server on controller1:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer 2019-06-05 18:50:33.217 1169 ERROR oslo.messaging._drivers.impl_rabbit [-] [bd6900e0-ab7b-4139-920c-a456d7df023b] AMQP server on controller1:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <RecoverableConnectionError: unknown error> The relevant service version numbers are as follow: rabbitmq-server-3.6.5-1.el7.noarch openstack-neutron-12.0.6-1.el7.noarch python2-oslo-messaging-5.35.4-1.el7.noarch Rabbitmq does not show any alert. It also has plenty of memory and a high enough file limit. The login user and credentials are fine as they are used in other openstack services which can contact rabbitmq without issues. I’ve tried optimizing rabbitmq, upgrading, downgrading, increasing timeouts in neutron services, etc, to no avail. I find myself at a loss and would appreciate if anyone has any idea as to where to go from there. Best regards, Jean-Philippe Méthot Openstack system administrator Administrateur système Openstack PlanetHoster inc.

4 6

[all] [TC] Call for Election Officials
by Kendall Nelson 10 Jun '19

10 Jun '19

Hello, The upcoming round for OpenStack's technical elections (both PTL and TC) is set to happen in September this year[1]. We would like to encourage anyone interested to volunteer to help administer the elections, especially as some of our long-service volunteers periodically become ineligible by way of being nominated for election themselves. While elections have historically been handled by a small number of volunteers, the intention was for this to NOT be a closed process. The election process is detailed in this document [2], with tooling managed in gerrit [3], and uses StoryBoard [4] to keep track of various election activities. We are happy to mentor individuals and share knowledge about the elections process in an effort to get more of the community involved. Involvement can be on an ongoing basis or simply volunteering to help with some small part of one election just to learn how the process works. The election officials team is explicitly delegated by the Technical Committee, who have historically consistently expressed interest in more volunteers to assist. Please let us know if you would like to volunteer! -Kendall Nelson & The Election Officials [1] https://review.opendev.org/#/c/661673/2 [2] https://governance.openstack.org/election/process.html [3] https://opendev.org/openstack/election [4] https://storyboard.openstack.org/#!/project/openstack/election

1 0

[tc][all] Train Community Goals
by Lance Bragstad 10 Jun '19

10 Jun '19

Hi all, The goals for the Train development cycle have merged. Both are available on governance.openstack.org [0][1]. Please have a look if you haven't already. Goal champions are asettle and gmann. As always, if you have any comments, questions, or concerns, please don't hesitate to reach out on the mailing list or in #openstack-tc. Thanks, Lance [0] https://governance.openstack.org/tc/goals/train/pdf-doc-generation.html [1] https://governance.openstack.org/tc/goals/train/ipv6-support-and-testing.ht…

1 1

[neutron] Bug deputy report for week of June 3rd
by Brian Haley 10 Jun '19

10 Jun '19

Hi, I was Neutron bug deputy last week. Below is a short summary about reported bugs. -Brian Critical bugs ------------- * https://bugs.launchpad.net/neutron/+bug/1832225 - Neutron-vpnaas unit tests broken - Broken by https://review.opendev.org/#/c/653903/ - https://review.opendev.org/#/c/664257/ High bugs --------- * https://bugs.launchpad.net/neutron/+bug/1831534 - [l3][dvr] with openflow security group east-west traffic between different vlan networks is broken - https://review.opendev.org/#/c/662925/ - https://review.opendev.org/#/c/663008/ * https://bugs.launchpad.net/neutron/+bug/1831575 - br-tun gets a wrong arp drop rule when dvr is connected to a network but not used as gateway - https://review.opendev.org/#/c/662999/ - https://review.opendev.org/#/c/663000/ * https://bugs.launchpad.net/neutron/+bug/1831647 - Creation of existing resource takes too much time or fails - https://review.opendev.org/#/c/663749/ * https://bugs.launchpad.net/neutron/+bug/1831919 - Impossible to change a list of static routes defined for subnet because of InvalidRequestError with Cisco ACI integration - https://review.opendev.org/#/c/663714/ - https://review.opendev.org/#/c/663713/ - https://review.opendev.org/#/c/663712/ Medium bugs ----------- * https://bugs.launchpad.net/neutron/+bug/1831404 - rarp packet will be dropped in flows cause vm connectivity broken after live-migration - Yang Li took ownership * https://bugs.launchpad.net/neutron/+bug/1831613 - SRIOV: agent may not register VFs - https://review.opendev.org/#/c/663031/ * https://bugs.launchpad.net/neutron/+bug/1831706 - [DVR] Modify `in_port` field of packets which from remote qr-* port - Possibly related to https://review.opendev.org/#/c/639009 and https://bugs.launchpad.net/neutron/+bug/1732067 * https://bugs.launchpad.net/neutron/+bug/1831811 - Unable to filter using same cidr value as used for subnet create - https://review.opendev.org/#/c/663464/ * https://bugs.launchpad.net/ubuntu/+source/neutron-fwaas/+bug/1832210 - incorrect decode of log prefix under python 3 - https://review.opendev.org/#/c/664234/ Low bugs -------- * https://bugs.launchpad.net/neutron/+bug/1831916 - BGP dynamic routing in neutron - https://review.opendev.org/#/c/663711/ Wishlist bugs ------------- None Invalid bugs ------------ * https://bugs.launchpad.net/bugs/1831613 - moved to lbaas storyboard Further triage required ----------------------- * https://bugs.launchpad.net/neutron/+bug/1831726 - neutron-cli port-update ipv6 fixed_ips Covering previous - Fixed IP getting replaced when using neutronclient, as API ref says will happen. - Looks like the openstackclient is doing things properly by appending the new fixed IP to the existing. - Might just be a documentation issue. * https://bugs.launchpad.net/neutron/+bug/1832021 - Checksum drop of metadata traffic on isolated provider networks - Related to recent revert of TCP checksum-fill iptables rule, https://review.opendev.org/#/c/654645/ - but since that was an invalid rule there is probably another issue here.

1 0

[placement] update 19-22
by Chris Dent 10 Jun '19

10 Jun '19

HTML: https://anticdent.org/placement-update-19-22.html Welcome to placement update 19-22. # Most Important We are continuing to work through issues associated with the [spec for nested magic](https://review.opendev.org/662191). Unsurprisingly, there are edge cases where we need to be sure we're doing the right thing, both in terms of satisfying the use cases as well as making sure we don't violate the general model of how things are supposed to work. # What's Changed * We've had a few responses on [the thread to determine the fate of can_split](http://lists.openstack.org/pipermail/openstack-discuss/2019-May/…. The consensus at this point is to not worry about workloads that mix NUMA-aware guests with non-NUMA-aware on the same host. * Support forbidden traits (microversion 1.22) has been added to osc-placement. * Office hours will be 1500 UTC on Wednesdays. * os-traits 0.13.0 and 0.14.0 were released. * Code to optionaly run a [wsgi profiler](https://docs.openstack.org/placement/latest/contributor/testing.h… in placement has merged. * The request group mapping in allocation candidates spec has merged, more on that in themes, below. # Specs/Features * <https://review.opendev.org/654799> Support Consumer Types. This has some open questions that need to be addressed, but we're still go on the general idea. * <https://review.opendev.org/662191> Spec for nested magic 1. The easier parts of nested magic: same_subtree, resource request groups, verbose suffixes (already merged as 1.33). Recently some new discussion here. These and other features being considered can be found on the [feature worklist](https://storyboard.openstack.org/#!/worklist/594). Some non-placement specs are listed in the Other section below. # Stories/Bugs (Numbers in () are the change since the last pupdate.) There are 20 (1) stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 (0) are [untagged](https://storyboard.openstack.org/#!/worklist/580). 3 (1) are [bugs](https://storyboard.openstack.org/#!/worklist/574). 4 (0) are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 11 (0) are [rfes](https://storyboard.openstack.org/#!/worklist/594). 2 (0) are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 15 (-1). * Placement related nova [in progress bugs](https://goo.gl/vzGGDQ) on launchpad: 7 (0). # osc-placement osc-placement is currently behind by 11 microversions. Pending Changes: * <https://review.openstack.org/#/c/640898/> Add 'resource provider inventory update' command (that helps with aggregate allocation ratios). * <https://review.openstack.org/586056> Provide a useful message in the case of 500-error # Main Themes ## Nested Magic The overview of the features encapsulated by the term "nested magic" are in a [story](https://storyboard.openstack.org/#!/story/2005575). There is some in progress code, some of it WIPs to expose issues: * <https://review.opendev.org/663009> PoC: resourceless request, including some code from <https://review.opendev.org/657510> WIP: Allow RequestGroups without resources * <https://review.opendev.org/657463> Add NUMANetworkFixture for gabbits * <https://review.opendev.org/662785> Prepare objects for allocation request mappings. This work exposed a [bug in hash handling](https://storyboard.openstack.org/#!/story/2005822) that is [being fixed](https://review.opendev.org/663137). ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. A [spec](https://review.opendev.org/654799) has started. There are some questions about request and response details that need to be resolved, but the overall concept is sound. ## Cleanup We continue to do cleanup work to lay in reasonable foundations for the nested work above. As a nice bonus, we keep eking out additional performance gains too. * <https://review.opendev.org/661769> Add olso.middleware.cors to conf generator * <https://review.opendev.org/661922> Modernize CORS config and setup. Ed Leafe's ongoing work with using a graph database probably needs some kind of report or update. # Other Placement Miscellaneous changes can be found in [the usual place](https://review.opendev.org/#/q/project:openstack/placement+status:op…. There are several [os-traits changes](https://review.opendev.org/#/q/project:openstack/os-traits+status:… being discussed. # Other Service Users New discoveries are added to the end. Merged stuff is removed. Anything that has had no activity in 4 weeks has been removed. * <https://review.openstack.org/601596> Nova: spec: support virtual persistent memory * <https://review.openstack.org/#/q/topic:bug/1819923> Nova: nova-manage: heal port allocations * <https://review.openstack.org/650188> nova-spec: Allow compute nodes to use DISK_GB from shared storage RP * <https://review.opendev.org/659233> Cyborg: Placement report * <https://review.opendev.org/657884> Nova: Spec to pre-filter disabled computes with placement * <https://review.opendev.org/657801> rpm-packaging: placement service * <https://review.opendev.org/657016> Delete resource providers for all nodes when deleting compute service * <https://review.opendev.org/654066> nova fix for: Drop source node allocations if finish_resize fails * <https://review.opendev.org/656885> nova: WIP: Hey let's support routed networks y'all! * <https://review.opendev.org/662371> starlingx: Add placement chart patch to openstack-helm * <https://review.opendev.org/662229> helm: WIP: add placement chart * <https://review.opendev.org/662164> kolla-ansible: Add a explanatory note for "placement_api_port" * <https://review.opendev.org/658451> neutron-spec: L3 agent capacity and scheduling * <https://review.opendev.org/656023> Nova: Use OpenStack SDK for placement * <https://review.opendev.org/659302> puppet: Implement generic placement::config::placement_config * <https://review.opendev.org/663272> puppet: Add parameter for `scheduler/query_placement_for_image_type_support` * <https://review.opendev.org/612497> Nova: Spec: Provider config YAML file * <https://review.opendev.org/623558> Nova: single pass instance info fetch in host manager * <https://review.opendev.org/661245> Watcher: Add Placement helper * <https://review.opendev.org/663478> docs: Add Placement service to Minimal deployment for Stein * <https://review.opendev.org/663270> devstack: Add setting of placement microversion on tempest conf * <https://review.opendev.org/634551> libvirt: report pmem namespaces resources by provider tree * <https://review.opendev.org/657796> Nova: Defaults missing group_policy to 'none' * <https://review.opendev.org/660852> Nova: Remove PlacementAPIConnectFailure handling from AggregateAPI # End Making good headway. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent

2 1

Cinder Ceph backup concurrency
by Cory Hawkless 10 Jun '19

10 Jun '19

I'm using Rocky and Cinders built in Ceph backup driver which is working ok but I'd like to limit each instance of the backup agent to X number of concurrent backups. For example, if I(Or a tenant) trigger a backup to run on 20 volumes, the cinder-0backuip agent promptly starts the process of backup up all 20 volumes simultaneously and while this works ok it has the downside of over saturating links, causing high IO on the disks etc. Ideally I'd like to have each cinder-backup agent limited to running X(Perhaps 5) backups jobs at any one time and the remaining jobs will be 'queued' until an agent has less than X jobs remaining. Is this possible at all? Based on my understanding the Cinder scheduler services handles the allocation and distribution of the backup tasks, is that correct? Thanks in advance Cory

3 2

[kolla] Feedback request: removing kolla-cli
by Mark Goddard 10 Jun '19

10 Jun '19

Hi, We discussed during the kolla virtual PTG [1] the option of removing support for the kolla-cli deliverable [2], as a way to improve the long term sustainability of the project. kolla-cli was a project started by Oracle, and accepted as a kolla deliverable. While it looks interesting and potentially useful, it never gained much traction (as far as I'm aware) and the maintainers left the community. We have never released it and CI has been failing for some time. We propose dropping support for kolla-cli in the Train cycle. If this will affect you and you would like to help maintain it, please get in touch. Thanks, Mark [1] https://etherpad.openstack.org/p/kolla-train-ptg [2] https://github.com/openstack/kolla-cli

1 1

[kolla] Feedback request: removing OracleLinux support
by Mark Goddard 10 Jun '19

10 Jun '19

Hi, We discussed during the kolla virtual PTG [1] the option of removing support for Oracle Linux, as a way to improve the long term sustainability of the project. Since (from afar) OracleLinux is very similar to CentOS, it does not require too much maintenance, however it is non-zero and does consume CI resources. Contributors from Oracle left the community some time ago, and we do not generally see Oracle Linux in bug reports, so must assume it is not well used. We propose dropping support for OracleLinux in the Train cycle. If this will affect you and you would like to help maintain it, please get in touch. Thanks, Mark [1] https://etherpad.openstack.org/p/kolla-train-ptg

2 2