- openstack-discuss - lists.openstack.org

[qa][infra][stable] Stable branches gate status: tempest-full-* jobs failing for stable/ocata|pike|queens
by Ghanshyam Mann 09 Jan '20

09 Jan '20

Hello Everyone, tempest-full-* jobs are failing on stable/queens, stable/pike, and stable/ocata(legacy-tempest-dsvm-neutron-full-ocata) [1].ld Please hold any recheck till fix is merged. whoami-rajat reported about the tempest-full-queens-py3 job failure and later while debugging we found that same is failing for pike and ocata(job name there - legacy-tempest-dsvm-neutron-full-ocata). Failure is due to "Timeout on connecting the vnc console url" because there is no 'n-cauth' service running which is required for these stable branches. In Ussuri that service has been removed from nova. 'n-cauth' has been removed from ENABLED_SERVICES recently in - https://review.opendev.org/#/c/700217/ which effected only stable branches till queens. stable/rocky|stein are working because we have moved the services enable things from devstack-gate's test matrix to devstack base job[2]. Patch[2] was not backported to stable/queens and stable/pike which I am not sure why. We have two ways to fix the stable branches gate: 1. re-enable the n-cauth in devstack-gate. Hope all other removes services create no problem. pros: easy to fix, fix for all three stable branches. patch- https://review.opendev.org/#/c/701404/ 2. Backport the 546765[2] to stable/queens and stable/pike. pros: this removes the dependency form test-matrix which is the overall goal to remove d-g dependency. cons: It cannot be backported to stable/ocata as no zuulv3 base jobs there. This is already EM and anyone still cares about this? I think for fixing the gate (Tempest master and stable/queens|pike|ocata), we can go with option 1 and later we backport the devstack migration. [1] - http://zuul.openstack.org/builds?job_name=tempest-full-queens-py3 - http://zuul.openstack.org/builds?job_name=tempest-full-pike - http://zuul.openstack.org/builds?job_name=legacy-tempest-dsvm-neutron-full-… - reported bug - https://bugs.launchpad.net/devstack/+bug/1858666 [2] https://review.opendev.org/#/c/546765/ -gmann

1 1

[tripleo] tripleo-operator-ansible start and request for input
by Alex Schultz 09 Jan '20

09 Jan '20

[Hello folks, I've begun the basic start of the tripleo-operator-ansible collection work[0]. At the start of this work, I've chosen the undercloud installation[1] as the first role to use to figure out how we the end user's to consume these roles. I wanted to bring up this initial implementation so that we can discuss how folks will include these roles. The initial implementation is a wrapper around the tripleoclient command as run via openstackclient. This means that the 'tripleo-undercloud' role provides implementations for 'openstack undercloud backup', 'openstack undercloud install', and 'openstack undercloud upgrade'. In terms of naming conventions, I'm proposing that we would name the roles "tripleo-<command-base>" with the last part of the command action being an "action". Examples: "openstack undercloud *" -> role: tripleo-undercloud action: (backup|install|upgrade) "openstack undercloud minion *" -> role: tripleo-undercloud-minion action: (install|upgrade) "openstack overcloud *" -> role: tripleo-overcloud action: (deploy|delete|export) "openstack overcloud node *" -> role: tripleo-overcloud-node action: (import|introspect|provision|unprovision) In terms of end user interface, I've got two proposals out in terms of possible implementations. Tasks from method: The initial commit propose that we would require the end user to use an include_role/tasks_from call to perform the desired action. For example: - hosts: undercloud gather_facts: true tasks: - name: Install undercloud collections: - tripleo.operator import_role: name: tripleo-undercloud tasks_from: install vars: tripleo_undercloud_debug: true Variable switch method: I've also proposed an alternative implementation[2] that would use include_role but require the end user to set a specific variable to change if the role runs 'install', 'backup' or 'upgrade'. With this patch the playbook would look something like: - hosts: undercloud gather_facts: true tasks: - name: Install undercloud collections: - tripleo.operator import_role: name: tripleo-undercloud vars: tripleo_undercloud_action: install tripleo_undercloud_debug: true I would like to solicit feedback on which one of these is the preferred integration method when calling these roles. I have two patches up in tripleo-quickstart-extras to show how these calls could be run. The "Tasks from method" can be viewed here[3]. The "Variable switch method" can be viewed here[4]. I can see pros and cons for both methods. My take would be: Tasks from method: Pros: - action is a bit more explicit - dynamic logic left up to the playbook/consumer. - May not have a 'default' action (as main.yml is empty, though it could be implemented). - tasks_from would be a global implementation across all roles rather than having a changing variable name. Cons: - internal task file names must be known by the consumer (though IMHO this is no different than the variable name + values in the other implementation) - role/action inclusions is not dynamic in the role (it can be in the playbook) Variable switch method: Pros: - inclusion of the role by default runs an install - action can be dynamically changed from the calling playbook via an ansible var - structure of the task files is internal to the role and the user of the role need not know the filenames/structure. Cons: - calling playbook is not explicit in that the action can be switched dynamically (e.g. intentionally or accidentally because it is dynamic) - implementer must know to configure a variable called `tripleo_undercloud_action` to switch between install/backup/upgrade actions - variable names are likely different depending on the role My personal preference might be to use the "Tasks from method" because it would lend itself to the same implementation across all roles and the dynamic logic is left to the playbook rather than internally in the role. For example, we'd end up with something like: - hosts: undercloud gather_facts: true collections: - tripleo.operator tasks: - name: Install undercloud import_role: name: tripleo-undercloud tasks_from: install vars: tripleo_undercloud_debug: true - name: Upload images import_role: name: tripleo-overcloud-images tasks_from: upload vars: tripleo_overcloud_images_debug: true - name: Import nodes import_role: name: tripleo-overcloud-node tasks_from: import vars: tripleo_overcloud_node_debug: true tripleo_overcloud_node_import_file: instack.json - name: Introspect nodes import_role: name: tripleo-overcloud-node tasks_from: introspect vars: tripleo_overcloud_node_debug: true tripleo_overcloud_node_introspect_all_manageable: True tripleo_overcloud_node_introspect_provide: True - name: Overcloud deploy import_role: name: tripleo-overcloud tasks_from: deploy vars: tripleo_overcloud_debug: true tripleo_overcloud_deploy_environment_files: - /home/stack/params.yaml The same general tasks performed via the "Variable switch method" would look something like: - hosts: undercloud gather_facts: true collections: - tripleo.operator tasks: - name: Install undercloud import_role: name: tripleo-undercloud vars: tripleo_undercloud_action: install tripleo_undercloud_debug: true - name: Upload images import_role: name: tripleo-overcloud-images vars: tripleo_overcloud_images_action: upload tripleo_overcloud_images_debug: true - name: Import nodes import_role: name: tripleo-overcloud-node vars: tripleo_overcloud_node_action: import tripleo_overcloud_node_debug: true tripleo_overcloud_node_import_file: instack.json - name: Introspect nodes import_role: name: tripleo-overcloud-node vars: tripleo_overcloud_node_action: introspect tripleo_overcloud_node_debug: true tripleo_overcloud_node_introspect_all_manageable: True tripleo_overcloud_node_introspect_provide: True - name: Overcloud deploy import_role: name: tripleo-overcloud vars: tripleo_overcloud_action: deploy tripleo_overcloud_debug: true tripleo_overcloud_deploy_environment_files: - /home/stack/params.yaml Thoughts? Thanks, -Alex [0] https://blueprints.launchpad.net/tripleo/+spec/tripleo-operator-ansible [1] https://review.opendev.org/#/c/699311/ [2] https://review.opendev.org/#/c/701628/ [3] https://review.opendev.org/#/c/701034/ [4] https://review.opendev.org/#/c/701628/

7 7

[infra] Retire openstack/js-openstack-lib repository
by Andreas Jaeger 09 Jan '20

09 Jan '20

The js-openstack-lib repository is orphaned and has not seen any real merges or contributions since February 2017, I propose to retire it. I'll send retirement changes using topic retire-js-openstack-lib, Andreas -- Andreas Jaeger aj(a)suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB

5 10

[largescale-sig] Meeting summary and next actions
by Thierry Carrez 09 Jan '20

09 Jan '20

Hi everyone, The Large Scale SIG held a meeting earlier today. You can access the summary and logs of the meeting at: http://eavesdrop.openstack.org/meetings/large_scale_sig/2019/large_scale_si… This meeting was focused on immediate next steps on the two initial goals for the SIG. For the "Scaling within one cluster, and instrumentation of the bottlenecks" goal, after the holidays we will collect user stories of what happens (or what breaks first) as you scale up single clusters. We will also produce a first RFC draft for an oslo.metric blueprint, and learn more about the "Golden signals" concept. Standing TODOs: - all prepare a short description of what happens (what breaks first) when scaling up a single cluster - ttx to prepare etherpad and send ML thread asking for user scaling stories, to be posted after the end-of-year holidays - masahito to produce first draft for the oslo.metric blueprint - all learn more about golden signals concept as described in https://landing.google.com/sre/book.html For the "Document large scale configuration and tips &tricks" goal we will start collecting relevant articles and blogposts. We agreed that in order to properly document large scale configuration defaults, we should probably both patch existing docs and create new documentation. Standing TODOs: - all add links to relevant articles around large scale openstack (if you find any) to the etherpad - oneswig to follow up with Scientific community to find such articles - amorin to start a thread on documenting configuration defaults for large scale, introduce the "mixture of both" tactic The next meeting will happen on January 15, at 9:00 UTC on #openstack-meeting. Cheers, -- Thierry Carrez (ttx)

3 4

[scientific][www] Unable to download OpenStack for Scientific Research book
by Pierre Riteau 09 Jan '20

09 Jan '20

Hello, I tried to download the book at https://www.openstack.org/science/ but the link doesn't work. Could this please be fixed? I looked on openstack.org for a contact address, but couldn't find one. Please let me know if there is a specific address I should use next time. Thanks, Pierre

2 2

[tripleo] Use Podman 1.6 in CI
by Luke Short 09 Jan '20

09 Jan '20

Hey folks, We have been running into a situation where an older version of Podman is used in CI that has consistent failures. It has problems deleting storage associated with a container. A possible workaround (originally created by Damien) can be found here [1]. A few of us been in talks with the Podman team about this problem and have tested with a newer version of it (1.6.4, to be exact) and found that it is no longer an issue. The most ideal situation is to simply use this newer version of Podman instead of adding hacky workarounds that we will soon revert. However, I am unsure about how we would go about doing this upstream. RHEL will soon get an updated Podman version but CentOS always lags behind. Even CentOS 8 Stream does not contain the newer version [2]. The question/ask I have is can we ship/use a newer version of Podman in our upstream CI? Or should we continue our efforts on making a workaround? 1. https://review.opendev.org/#/c/698999/ 2. http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/ Sincerely, Luke Short

5 6

[Multi-Arch SIG] summary and actions from last meeting
by Rico Lin 09 Jan '20

09 Jan '20

Hi all Thanks to all who sign up and help to form Multi-Arch SIG We hosted our two initial meetings this week [1] and both successful. And for who cares about multi-arch (for example like ARM support in community), feel free to join our future meetings [2]. Here are some actions from this week's meetings: * Create StoryBoard for Multi-Arch SIG (ricolin) * Build multi-arch SIG repo (ricolin) * Collect update ppc64le actions and resources in community if see any (mrda) * Help with documentations once Multi-Arch SIG repo is ready (jeremyfreudberg) * Update governance-sigs to make a more clear description for Multi-Arch SIG (jeremyfreudberg) There are two docs. ideas `oh, on arm64 you need to do XYZ in other way - here is how and why` and `use cases, who, and issues they had`. Both of them seem like great docs. to start with. So that I assume will be something this SIG will try to work on as one of our first step goals. We need more people to help and more resource in CI as well. There's a lot of WIP resources and CI jobs in our community and all of them are collected in our etherpad [3] now. Feel free to update that etherpad and also sign you or your organization's name on it. Please join us:) Once we have our StoryBoard ready, we should be able to create tasks on it so everyone can create and track tasks like CI job, documentations, etc. last but not least, we are looking for people to nominate or volunteer for chair roles (which can be multiple). There are some really experienced community members been nominated and I will check with them if willing to take the role. On the other hand, I'm volunteering to apply for one of SIG chair seats and help to build this SIG, but happy to give it to others if we can have more people sign up for that role:) So let me know if you're interested. [1] http://eavesdrop.openstack.org/meetings/multi_arch/2020/ [2] http://eavesdrop.openstack.org/#Multi-Arch_SIG_Meeting [3] https://etherpad.openstack.org/p/Multi-arch -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

1 0

[cinder] ussuri virtual mid-cycle poll
by Brian Rosmaita 08 Jan '20

08 Jan '20

As determined at the virtual PTG (which seems like it happened only a few weeks ago), we'll be doing a two-phase virtual mid-cycle, meeting for two hours the week of 20 January (before the spec freeze) and again around the week of 16 March (the Cinder new feature status checkpoint). There's a poll up to determine a suitable time for the first virtual mid-cycle meeting: https://doodle.com/poll/n3tmq8ep43dyi7tv Please fill out the poll as soon as you can. If all the times are horrible for you, please suggest an alternative in a comment on the poll. The poll will close at 23:59 UTC on Saturday 11 January. (I know it's soon, but that way we'll have time to make adjustments if necessary.) cheers, brian

1 0

[all][infra] Compressed job log artifacts not loading in browser
by Clark Boylan 08 Jan '20

08 Jan '20

Over the holidays the infra team noticed that some of our release artifacts were in the wrong file format. On further investigation we discovered the reason for this was some swift implementations were inflating compressed tarballs when retrieved by clients not setting accept-encoding: gzip. We then ended up with tar files when we expected .tar.gz format files. This behavior seems to be controlled by the content-encoding we set on object upload. If we tell swift that the object is a gzip'd file on upload then the webservers helpfully inflate them when a client retrieves them. In order to fix our release artifacts we have updated our swift upload tooling to stop setting content-type on gzip files. This forces the swift implementation to return the files in the same format they are uploaded when retrieved. A side effect of this is that any gzip'd files (like testr_results.html.gz) are no longer automatically decompressed for you when you retrieve them. We have fixes up to handle common occurrences of this at https://review.opendev.org/#/c/701578/ and https://review.opendev.org/701282 (note the second case is already handled on OpenDev's Zuul). If you run into other files you expect to be browesable but get compressed instead, the fix is to stop compressing the files explicitly in the job. We will still upload files in compressed form to swift for efficiency but we should operate on these files as if they were uncompressed. Then for files that should be compressed (like .tar.gz files) we can pass them through as is avoiding any format change problems. Clark

2 1

[cliff][docs][requirements] new cliff versions causes docs to fail to build
by Matthew Thode 08 Jan '20

08 Jan '20

Looks like some things changed in the new version that we depended upon and are now causing failures. Exception occurred: File "/home/zuul/src/opendev.org/openstack/python-openstackclient/.tox/docs/lib/python3.6/site-packages/cliff/sphinxext.py", line 245, in _load_app if not issubclass(cliff_app_class, app.App): TypeError: issubclass() arg 1 must be a class -- Matthew Thode

4 6

[Manila] First meeting of 2020
by Tom Barron 08 Jan '20

08 Jan '20

Hey Zorillas! Just a reminder of our first meeting in 2020, 9 January at 1500 UTC on Freenode #openstack-meeting-alt. Feel free to update the agenda [1]. -- Tom [1] https://wiki.openstack.org/wiki/Manila/Meetings#Next_meeting

1 0

Compute node NIC bonding for increased instance throughput
by shubjero 08 Jan '20

08 Jan '20

Good day, I have a question for the OpenStack community, hopefully someone can help me out here. Goal ------------------ Provision an NFS instance capable of providing 20Gbps of network throughput to be used by multiple other instances within the same project/network. Background ------------------ We run an OpenStack Stein cluster on Ubuntu 18.04. Our Neutron architecture is using openvswitch and GRE. Our compute nodes have two 10G NIC's and are configured in a layer3+4 LACP to the Top of Rack switch. Observations ------------------ Successfully see 20Gbps of traffic balanced across both slaves in the bond when performing iperf3 tests at the *baremetal/os/ubuntu* layer with two other compute nodes as iperf3 clients. Problem ------------------ We are unable to achieve 20Gbps at the instance level. We have tried multiple iperf3 connections from multiple other instances on different compute nodes and we are only able to reach 10Gbps and notice that traffic is not utilizing both slaves in the bond. One slave gets all of the traffic while the other slave sits basically idle. I have some configuration output here: http://paste.openstack.org/show/QdQq76q6VI1XN5tLW0xH/ Any help would be appreciated! Jared Baker Cloud Architect, Ontario Institute for Cancer Research

2 1

Re: OIDC/OAuth2 token introspection in Keystone
by Nikolla, Kristi 08 Jan '20

08 Jan '20

There is an patch to improve the documentation for using the CLI with OIDC, but it hasn't merged yet. See here https://review.opendev.org/#/c/693838 Keystoneauth has plugins in place for authenticating with the OIDC IdP in multiple ways, including using an access token, see here https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/identit… Best, Kristi On Jan 8, 2020, at 10:31 AM, mcarpene <m.carpen(a)cineca.it<mailto:m.carpen@cineca.it>> wrote: Many thanks Nikolla, I was able to federate using OIDC IdP via the dashboard. I meant the problem is authenticating via CLI providing a OIDC token via command line, but maybe you already answered to my request. BR, Michele On 08/01/20 16:28, wrote: Hi Michele, We just approved a feature request for that [0], however it was merged to backlog, meaning no specific timeline for it being implemented yet. With the current implementation, you can use OAuth 2.0 Access Tokens with Keystone, however the token introspection endpoint will be used, therefore only the claims contained in the access token will be returned. I am assuming your question is with regards to the userinfo endpoint and OIDC claims, which we do not currently support. [0]. https://review.opendev.org/#/c/373983/ On Jan 8, 2020, at 8:01 AM, mcarpene <m.carpen(a)cineca.it<mailto:m.carpen@cineca.it>> wrote: Hi all, my question is: could OS Keystone support OIDC/OAuth2 token introspection/validation. I mean for example executing a swift command via CLI adding a OIDC token bearer as a parameter to the swift command. In this case Keystone should validate the OIDC token towards and external IdP (using introspection endpoint/protocol for oidc). Is this currently supported, or eventually would be done in the near future? thanks Michele -- Michele Carpené SuperComputing Applications and Innovation Department CINECA - via Magnanelli, 6/3, 40033 Casalecchio di Reno (Bologna) - ITALY Tel: +39 051 6171730 Fax: +39 051 6132198 Skype: mcarpene http://www.hpc.cineca.it/ -- Michele Carpené SuperComputing Applications and Innovation Department CINECA - via Magnanelli, 6/3, 40033 Casalecchio di Reno (Bologna) - ITALY Tel: +39 051 6171730 Fax: +39 051 6132198 Skype: mcarpene http://www.hpc.cineca.it/

1 0

OIDC/OAuth2 token introspection in Keystone
by mcarpene 08 Jan '20

08 Jan '20

Hi all, my question is: could OS Keystone support OIDC/OAuth2 token introspection/validation. I mean for example executing a swift command via CLI adding a OIDC token bearer as a parameter to the swift command. In this case Keystone should validate the OIDC token towards and external IdP (using introspection endpoint/protocol for oidc). Is this currently supported, or eventually would be done in the near future? thanks Michele -- Michele Carpené SuperComputing Applications and Innovation Department CINECA - via Magnanelli, 6/3, 40033 Casalecchio di Reno (Bologna) - ITALY Tel: +39 051 6171730 Fax: +39 051 6132198 Skype: mcarpene http://www.hpc.cineca.it/

2 1

[charms][watcher] OpenStack Watcher Charm
by Stamatis Katsaounis 08 Jan '20

08 Jan '20

Hi all, Purpose of this email is to let you know that we released an unofficial charm of OpenStack Watcher [1]. This charm gave us the opportunity to deploy OpenStack Watcher to our charmed OpenStack deployment. After seeing value in it, we decided to publish it through GRNET GitHub Organization account for several reasons. First of all, we would love to get feedback on it as it is our first try on creating an OpenStack reactive charm. Secondly, we would be glad to see other OpenStack operators deploy Watcher and share with us knowledge on the project and possible use cases. Finally, it would be ideal to come up with an official OpenStack Watcher charm repository under charmers umbrella. By doing this, another OpenStack project is going to be available not only for Train version but for any future version of OpenStack. Most important, the CI tests are going to ensure that the code is not broken and persuade other operators to use it. Before closing my email, I would like to give some insight on the architecture of the code base and the deployment process. To begin with, charm-watcher is based on other reactive OpenStack charms. During its deployment Barbican, Designate, Octavia and other charms' code bases were counseled. Furthermore, the structure is the same as any official OpenStack charm, of course without functional tests, which is something we cannot provide. Speaking about the deployment process, apart from having a basic charmed OpenStack deployment, operator has to change two tiny configuration options on Nova cloud controller and Cinder. As explained in the Watcher configuration guide, special care has to be done with Oslo notifications for Nova and Cinder [2]. In order to achieve that in charmed OpenStack some issues were met and solved with the following patches [3], [4], [5], [6]. With these patches, operator can set the extra Oslo configuration and this is the only extra configuration needs to take place. Finally, with [7] Keystone charm can accept a relation with Watcher charm instead of ignoring it. To be able to deploy GRNET Watcher charm on Train, patches [3], [4], [5] and [7] have to be back-ported to stable/19.10 branch but that will require the approval of charmers team. Please let me know if such an option is available and in that case I am going to open the relevant patches. Furthermore, if you think that it could be a good option to create a spec and then introduce an official Watcher charm, I would love to help on that. I wish all a happy new year and I am looking forward to your response and possible feedback. PS. If we could have an Ubuntu package for watcher-dashboard [8] like octavia-dashboard [9] we would release a charm for it as well. Best regards, Stamatis Katsaounis [1] https://github.com/grnet/charm-watcher [2] https://docs.openstack.org/watcher/latest/configuration/configuring.html#co… [3] https://review.opendev.org/#/c/699079/ [4] https://review.opendev.org/#/c/699081/ [5] https://review.opendev.org/#/c/699657/ [6] https://github.com/juju/charm-helpers/pull/405 [7] https://review.opendev.org/#/c/699082/ [8] https://github.com/openstack/watcher-dashboard [9] https://launchpad.net/ubuntu/+source/octavia-dashboard -- [cid:part10.101270FA.EDBB3A4A@admin.grnet.gr] Stamatis Katsaounis DevOps Engineer t :(+30) 210 7471130 (ext. 483) f : + 30 210 7474490 GRNET | Networking Research and Education www.grnet.gr<http://www.grnet.gr> | 7, Kifisias Av., 115 23, Athens [Facebook icon]<https://www.facebook.com/grnet.gr/> [Twitter icon] <https://twitter.com/grnet_gr> [Youtube icon] <https://www.youtube.com/user/EDETvideos> [LinkedIn icon] <https://www.linkedin.com/company/55267/> [Instagram icon] <https://www.instagram.com/grnet.gr/>

1 0

[neutron][ironic] dynamic routing protocol status for routed network
by 양유석 08 Jan '20

08 Jan '20

Hi, For ironic flat provider network, I found it's hard to scale manually (for many racks). So I'm trying to use routed network for the purpose. One blur thing for routed network is how we can handle segment connectivity. >From reference (https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html) there is future work about dynamic routing protocol though, I could not find any hints for the functionality. How do you guys using routed network handle the segments' connectivity? Is there any project to speak subnet info via BGP? Thanks in advance.

1 0

VM boot volume disappears from compute's multipath Daemon when NetApp controller is placed offline
by Ahmed ZAKY 07 Jan '20

07 Jan '20

I have a setup where each VM gets assigned two vDisks, one encrypted boot volume and another storage volume. Storage used is NetApp (tripleo-netapp). With two controllers on NetApp side working in active/ active mode. My test case goes as follows: - I stop one of the active controllers. - I stop one of my VMs using OpenStack server stop - I then start my VM one more time using OpenStack server start. - VM fails to start. Here're my findings, hope someone would help if they can explain me the behaviour seen below: My VM: vel1bgw01-MCM2, it is running on compute overcloud-sriovperformancecompute-3.localdomain [root@overcloud-controller-0 (vel1asbc01) cbis-admin]# openstack server show vel1bgw01-MCM2 +--------------------------------------+------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------------------------+------------------------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | zone1 | | OS-EXT-SRV-ATTR:host | overcloud-sriovperformancecompute-3.localdomain | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-sriovperformancecompute-3.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-00000 <+4400000>e93 | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2019-12-18T15:49:37.000000 <+4437000000> | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | SBC01_MGW01_TIPC=192.168.48.22 <+441921684822>; SBC01_MGW01_DATAPATH_MATE=192.168.16.11 <+441921681611>; SBC01_MGW01_DATAPATH=192.168.32.8 <+44192168328> | | config_drive | True | | created | 2019-12-18T15:49:16Z | | flavor | SBC_MCM (asbc_mcm) | | hostId | 7886 <+447886>df0f7a3d4e131304 <+44131304>a8eb860e6a704c5fda2a7ed751b544ff2bf5 | | id | 5c70a984-89 <+4498489>a9-44ce-876d-9e2e568eb819 | | image | | | key_name | CBAM-b5fd59a066e8450 <+448450>ca9f104a69da5a043-Keypair | | name | vel1bgw01-MCM2 | | os-extended-volumes:volumes_attached | [{u'id': u'717e5744-4786-42 <+445744478642>dc-9e3e-3c5e6994 <+446994>c482'}, {u'id': u'd6cf0cf9-36d1-4b 62-86 <+446286>b4-faa4a6642166 <+446642166>'}] | | progress | 0 | | project_id | 41777 <+4441777>c6f1e7b4f8d8fd76b5e0f67e5e8 | | properties | | | security_groups | [{u'name': u'vel1bgw01-TIPC-Security-Group'}] | | status | ACTIVE | | updated | 2020-01-07T17:18:32Z | | user_id | be13deba85794016 <+4485794016>a00fec9d18c5d7cf | +--------------------------------------+------------------------------------------------------------------------------------------------------------+ *It is mapped to the following vDisks (seen using virsh list on compute-3)*: - dm-uuid-mpath-3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714 <+440714> è Boot Volume - dm-uuid-mpath-3600 <+443600>a098000 <+44098000>d9818 <+449818>b 000018565 <+44000018565>dfa069e è Storage volume <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/disk/by-id/dm-uuid-mpath-3600 <+443600>a098000 <+44098000> d9818 <+449818>b0000185 <+440000185>c5dfa0714 <+440714>'/> <backingStore/> <target dev='vda' bus='virtio'/> <serial>717e5744-4786-42 <+445744478642>dc-9e3e-3c5e6994 <+446994> c482</serial> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000 <+440000>' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/disk/by-id/dm-uuid-mpath-3600 <+443600>a098000 <+44098000> d9818 <+449818>b000018565 <+44000018565>dfa069e'/> <backingStore/> <target dev='vdb' bus='virtio'/> <serial>d6cf0cf9-36d1-4b62-86 <+446286>b4-faa4a6642166 <+446642166></serial> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000 <+440000>' bus='0x00' slot='0x08' function='0x0'/> </disk> Name: crypt-dm-uuid-mpath-3600 <+443600>a098000 <+44098000>d9818 <+449818>b 0000185 <+440000185>c5dfa0714 <+440714> State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 5 Number of targets: 1 UUID: CRYPT-LUKS1-769 <+441769>cc20bc5af469c8c9075 <+449075> a2a6fc4aa0-crypt-dm-uuid-mpath-*3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714 <+440714>* Name: *mpathpy* State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 32 Major, minor: 253, *4* Number of targets: 1 UUID: mpath-*3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714 <+440714>* Name: crypt-dm-uuid-mpath-3600 <+443600>a098000 <+44098000>d9818 <+449818>b 000018565 <+44000018565>dfa069e State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 7 Number of targets: 1 UUID: CRYPT-LUKS1-4015 <+4414015>c585a0df4074821 <+444074821> ca312c4caacca-crypt-dm-uuid-mpath-*3600 <+443600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e* Name: *mpathpz* State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 28 Major, minor: 253, *6* Number of targets: 1 UUID: mpath-*3600 <+443600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e* This means boot volume is represented by dm-4 while storage volume is represented by dm-6 Dumping the multipath daemon on the controller shows that at a steady running state both DMs are accounted for (see below). multipathd> show maps name sysfs uuid *mpathpy dm-4 3600 <+4443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714 <+440714>* *mpathpz dm-6 3600 <+4463600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e* mpathqi dm-12 3600 <+44123600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>df5dfafd40 mpathqj dm-13 3600 <+44133600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>de5dfafd10 mpathpw dm-0 3600 <+4403600>a098000 <+44098000>d9818 <+449818>b000018425 <+44000018425>dfa059f mpathpx dm-1 3600 <+4413600>a098000 <+44098000>d9818 <+449818>b0000184 <+440000184>c5dfa05fc mpathqk dm-16 3600 <+44163600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>eb5dfafe80 mpathql dm-17 3600 <+44173600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>e95dfafe26 mpathqh dm-9 3600 <+4493600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>c65dfafa91 These vDisks are mapped to the following multipaths: multipathd> show topology mpathpy (3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714) <+440714> dm-4 NETAPP ,INF-01-00 <+440100> size=21G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw |-+- policy='service-time 0' prio=14 status=active | |- 30:0:0:82 sdm 8:192 active ready running | `- 32:0:0:82 sdk 8:160 active ready running `-+- policy='service-time 0' prio=0 status=enabled |- 33:0:0:82 sdn 8:208 failed faulty running `- 31:0:0:82 sdl 8:176 failed faulty running mpathpz (3600 <+443600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e) dm-6 NETAPP ,INF-01-00 <+440100> size=10G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw |-+- policy='service-time 0' prio=14 status=active | |- 30:0:0:229 sdr 65:16 active ready running | `- 32:0:0:229 sdp 8:240 active ready running `-+- policy='service-time 0' prio=0 status=enabled |- 31:0:0:229 sdo 8:224 failed faulty running `- 33:0:0:229 sdq 65:0 failed faulty running Now, it starts getting very interesting, if I shutdown controller-A from NetApp side, dm-4 disappears but dm-6 is still running while detecting the active path is controller B while standby path is controller-A which now is displayed as failed multipathd> show topology mpathpz (3600 <+443600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e) dm-6 NETAPP ,INF-01-00 <+440100> size=10G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw |-+- policy='service-time 0' prio=0 status=enabled | |- 30:0:0:229 sdr 65:16 failed faulty running | `- 32:0:0:229 sdp 8:240 failed faulty running `-+- policy='service-time 0' prio=11 status=active |- 31:0:0:229 sdo 8:224 active ready running `- 33:0:0:229 sdq 65:0 active ready running multipathd> show maps name sysfs uuid *mpathpz dm-6 3600 <+4463600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e* mpathqi dm-12 3600 <+44123600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>df5dfafd40 mpathqj dm-13 3600 <+44133600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>de5dfafd10 mpathpw dm-0 3600 <+4403600>a098000 <+44098000>d9818 <+449818>b000018425 <+44000018425>dfa059f mpathpx dm-1 3600 <+4413600>a098000 <+44098000>d9818 <+449818>b0000184 <+440000184>c5dfa05fc mpathqk dm-16 3600 <+44163600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>eb5dfafe80 mpathql dm-17 3600 <+44173600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>e95dfafe26 mpathqg dm-8 3600 <+4483600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>c75dfafac0 mpathqh dm-9 3600 <+4493600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>c65dfafa91 If I restore Controller-A into service from NetApp side while fail only the path to controller A from multipathd everything works fine, dm-4 is still present and the VM can be put into service. multipathd> fail path sdk ok multipathd> multipathd> fail path sdm ok mpathpy (3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714) <+440714> dm-4 NETAPP ,INF-01-00 <+440100> size=21G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw |-+- policy='service-time 0' prio=0 status=enabled | |- 32:0:0:82 sdk 8:160 failed faulty running | `- 30:0:0:82 sdm 8:192 failed faulty running `-+- policy='service-time 0' prio=9 status=active |- 31:0:0:82 sdl 8:176 active ready running `- 33:0:0:82 sdn 8:208 active ready running multipathd> reinstate path sdk ok multipathd> multipathd> reinstate path sdm ok mpathpy (3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714) <+440714> dm-4 NETAPP ,INF-01-00 <+440100> size=21G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw |-+- policy='service-time 0' prio=14 status=active | |- 32:0:0:82 sdk 8:160 active ready running | `- 30:0:0:82 sdm 8:192 active ready running `-+- policy='service-time 0' prio=9 status=enabled |- 31:0:0:82 sdl 8:176 active ready running `- 33:0:0:82 sdn 8:208 active ready running It is observed in the working case, the storage volume disappears (which seems normal), also the instance totally vanishes from the virsh list and no trace can be found at the KVM level if we run ps -def | grep fd | grep <instance_ID>. However, the boot volume is always present in the multipathd records when we stop the VM at normal conditions without stopping NetApp controller. Any ideas? Kind regards, Ahmed

1 0

[infra] Retire x/dox
by Andreas Jaeger 07 Jan '20

07 Jan '20

The x/dox repo is unused, let's retire it. I'll put up changes with topic "retire-dox", Andreas -- Andreas Jaeger aj(a)suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB

1 0

[cinder] first meeting of 2020 tomorrow (8 January)
by Brian Rosmaita 07 Jan '20

07 Jan '20

Just wanted to send a quick reminder that the first Cinder team meeting of 2020 well be held tomorrow (8 January) on the usual day (Wednesday) at the usual time (1400 UTC) and in the usual place (#openstack-meeting-4). https://etherpad.openstack.org/p/cinder-ussuri-meetings See you there! brian

1 0

IBM z/VM CI is planning to migrate new environment
by Dong JV Ma 07 Jan '20

07 Jan '20

Hello, Due to the hardware issues, the IBM z/VM CI system [1] is planning to migrate to the new hardware in the next few months, the planning time will be in 3 months. We will do following changes with IBM z/VM CI environment: 1. Migrate the IBM z/VM CI to new hardware 2. Upgrade IBM z/VM CI from python2 to python3 During this time, IBM z/VM CI will not stable, please ignore the IBM z/VM CI failures. For any questions feel free to reach out to me(Dong Ma). [1] https://wiki.openstack.org/wiki/ThirdPartySystems/IBM_z/VM_CI Thanks & Best Regards! Dong Ma (马冬) LinuxONE IaaS Infra & CI/CD, China Systems Lab E-mail: madongma(a)cn.ibm.com IRC: larainema Address: 2/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC

3 2

[tc] Expediting patch to unblock governance's CI
by Jean-Philippe Evrard 07 Jan '20

07 Jan '20

Hello everyone, Our governance repo's testing is currently failing, and I have patches to fix it (short term [1] and long term). Because it's been a while it's in there (and got recently updated), I will now merge the short term fix, even if we don't have enough votes (nor time). It will unblock many patches and prevent useless rechecks, at the expense of being out of policy. I will gladly take the blame, should there be any. Regards, JP [1] https://review.opendev.org/#/c/700422/

1 0

[kolla] neutron-l3-agent namespace NAT table not working?
by Jon Masters 07 Jan '20

07 Jan '20

Hi there, I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the iptables rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or SNAT applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING chains (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log entries. It's as if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is driving me crazy :) Anyone got some quick suggestions? (assume I tried the obvious stuff). Jon. -- Computer Architect

8 19

[Scientific] Scientific SIG meeting July 7th 2100 UTC
by Martial Michel 07 Jan '20

07 Jan '20

The first meeting of the new year at 2100 UTC on Tuesday, July 7th. Mostly an Any Other Business meeting to get its participants back from the holidays as reflected by the agenda :) https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_January_7th_2020

1 0

[all] Is there something I can do to get a simple fix done?
by Neil Jerram 06 Jan '20

06 Jan '20

I'm struggling to say this positively, but... it feels like OpenStack promotes refactoring work that will likely break something, but is very slow when a corresponding fix is needed, even when the fix is trivial. Is there something we could do to get fixes done more quickly when needed? My case in point: my team's networking plugin (networking-calico) does not do "extraroutes", and so was broken by some python-openstackclient change (possibly [1]) that wrongly assumed that. I posted a fix [2] that passed CI on 21st October, and asked for it to be reviewed on IRC a couple of days later. Édouard Thuleau posted a similar fix [3] on 4th December, and we agreed that his was better, so I abandoned mine. His fix attracted a +2 on 11th December, but has been sitting like that ever since. It's a fix that I would expect to be simple to review, so I wonder if there's something else we could have done here to get this moving? Or if there is a systemic problem here that deserves discussion? [1] https://opendev.org/openstack/python-openstackclient/commit/c44f26eb7e41c28… [2] https://review.opendev.org/#/c/685312/ [3] https://review.opendev.org/#/c/697240/ Many thanks, Neil

2 1

[all][neutron][neutron-vpnaas] Maintainers needed
by Slawek Kaplonski 06 Jan '20

06 Jan '20

Hi, Over the past couple of cycles we have noticed that new contributions and maintenance efforts for neutron-vpnaas were almost non existent. This impacts patches for bug fixes, new features and reviews. The Neutron core team is trying to at least keep the CI of this project healthy, but we don’t have enough knowledge about the details of the neutron-vpnaas code base to review more complex patches. During the PTG in Shanghai we discussed that with operators and TC members during the forum session [1] and later within the Neutron team during the PTG session [2]. During these discussions, with the help of operators and TC members, we reached the conclusion that we need to have someone responsible for maintaining project. This doesn’t mean that the maintainer needs to spend full time working on this project. Rather, we need someone to be the contact person for the project, who takes care of the project’s CI and review patches. Of course that’s only a minimal requirement. If the new maintainer works on new features for the project, it’s even better :) If we don’t have any new maintainer(s) before milestone Ussuri-2, which is Feb 10 - Feb 14 according to [3], we will need to mark neutron-vpnaas as deprecated and in “V” cycle we will propose to move the project from the Neutron stadium, hosted in the “openstack/“ namespace, to the unofficial projects hosted in the “x/“ namespace. So if You are using this project now, or if You have customers who are using it, please consider the possibility of maintaining it. Otherwise, please be aware that it is highly possible that the project will be deprecated and moved out from the official OpenStack projects. [1] https://etherpad.openstack.org/p/PVG-Neutron-stadium-projects-the-path-forw… [2] https://etherpad.openstack.org/p/Shanghai-Neutron-Planning-restored - Lines 379-421 [3] https://releases.openstack.org/ussuri/schedule.html -- Slawek Kaplonski Senior software engineer Red Hat

1 1