- openstack-discuss - lists.openstack.org

[Manila] First meeting of 2020
by Tom Barron 08 Jan '20

08 Jan '20

Hey Zorillas! Just a reminder of our first meeting in 2020, 9 January at 1500 UTC on Freenode #openstack-meeting-alt. Feel free to update the agenda [1]. -- Tom [1] https://wiki.openstack.org/wiki/Manila/Meetings#Next_meeting

1 0

Compute node NIC bonding for increased instance throughput
by shubjero 08 Jan '20

08 Jan '20

Good day, I have a question for the OpenStack community, hopefully someone can help me out here. Goal ------------------ Provision an NFS instance capable of providing 20Gbps of network throughput to be used by multiple other instances within the same project/network. Background ------------------ We run an OpenStack Stein cluster on Ubuntu 18.04. Our Neutron architecture is using openvswitch and GRE. Our compute nodes have two 10G NIC's and are configured in a layer3+4 LACP to the Top of Rack switch. Observations ------------------ Successfully see 20Gbps of traffic balanced across both slaves in the bond when performing iperf3 tests at the *baremetal/os/ubuntu* layer with two other compute nodes as iperf3 clients. Problem ------------------ We are unable to achieve 20Gbps at the instance level. We have tried multiple iperf3 connections from multiple other instances on different compute nodes and we are only able to reach 10Gbps and notice that traffic is not utilizing both slaves in the bond. One slave gets all of the traffic while the other slave sits basically idle. I have some configuration output here: http://paste.openstack.org/show/QdQq76q6VI1XN5tLW0xH/ Any help would be appreciated! Jared Baker Cloud Architect, Ontario Institute for Cancer Research

2 1

Re: OIDC/OAuth2 token introspection in Keystone
by Nikolla, Kristi 08 Jan '20

08 Jan '20

There is an patch to improve the documentation for using the CLI with OIDC, but it hasn't merged yet. See here https://review.opendev.org/#/c/693838 Keystoneauth has plugins in place for authenticating with the OIDC IdP in multiple ways, including using an access token, see here https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/identit… Best, Kristi On Jan 8, 2020, at 10:31 AM, mcarpene <m.carpen(a)cineca.it<mailto:m.carpen@cineca.it>> wrote: Many thanks Nikolla, I was able to federate using OIDC IdP via the dashboard. I meant the problem is authenticating via CLI providing a OIDC token via command line, but maybe you already answered to my request. BR, Michele On 08/01/20 16:28, wrote: Hi Michele, We just approved a feature request for that [0], however it was merged to backlog, meaning no specific timeline for it being implemented yet. With the current implementation, you can use OAuth 2.0 Access Tokens with Keystone, however the token introspection endpoint will be used, therefore only the claims contained in the access token will be returned. I am assuming your question is with regards to the userinfo endpoint and OIDC claims, which we do not currently support. [0]. https://review.opendev.org/#/c/373983/ On Jan 8, 2020, at 8:01 AM, mcarpene <m.carpen(a)cineca.it<mailto:m.carpen@cineca.it>> wrote: Hi all, my question is: could OS Keystone support OIDC/OAuth2 token introspection/validation. I mean for example executing a swift command via CLI adding a OIDC token bearer as a parameter to the swift command. In this case Keystone should validate the OIDC token towards and external IdP (using introspection endpoint/protocol for oidc). Is this currently supported, or eventually would be done in the near future? thanks Michele -- Michele Carpené SuperComputing Applications and Innovation Department CINECA - via Magnanelli, 6/3, 40033 Casalecchio di Reno (Bologna) - ITALY Tel: +39 051 6171730 Fax: +39 051 6132198 Skype: mcarpene http://www.hpc.cineca.it/ -- Michele Carpené SuperComputing Applications and Innovation Department CINECA - via Magnanelli, 6/3, 40033 Casalecchio di Reno (Bologna) - ITALY Tel: +39 051 6171730 Fax: +39 051 6132198 Skype: mcarpene http://www.hpc.cineca.it/

1 0

OIDC/OAuth2 token introspection in Keystone
by mcarpene 08 Jan '20

08 Jan '20

Hi all, my question is: could OS Keystone support OIDC/OAuth2 token introspection/validation. I mean for example executing a swift command via CLI adding a OIDC token bearer as a parameter to the swift command. In this case Keystone should validate the OIDC token towards and external IdP (using introspection endpoint/protocol for oidc). Is this currently supported, or eventually would be done in the near future? thanks Michele -- Michele Carpené SuperComputing Applications and Innovation Department CINECA - via Magnanelli, 6/3, 40033 Casalecchio di Reno (Bologna) - ITALY Tel: +39 051 6171730 Fax: +39 051 6132198 Skype: mcarpene http://www.hpc.cineca.it/

2 1

[charms][watcher] OpenStack Watcher Charm
by Stamatis Katsaounis 08 Jan '20

08 Jan '20

Hi all, Purpose of this email is to let you know that we released an unofficial charm of OpenStack Watcher [1]. This charm gave us the opportunity to deploy OpenStack Watcher to our charmed OpenStack deployment. After seeing value in it, we decided to publish it through GRNET GitHub Organization account for several reasons. First of all, we would love to get feedback on it as it is our first try on creating an OpenStack reactive charm. Secondly, we would be glad to see other OpenStack operators deploy Watcher and share with us knowledge on the project and possible use cases. Finally, it would be ideal to come up with an official OpenStack Watcher charm repository under charmers umbrella. By doing this, another OpenStack project is going to be available not only for Train version but for any future version of OpenStack. Most important, the CI tests are going to ensure that the code is not broken and persuade other operators to use it. Before closing my email, I would like to give some insight on the architecture of the code base and the deployment process. To begin with, charm-watcher is based on other reactive OpenStack charms. During its deployment Barbican, Designate, Octavia and other charms' code bases were counseled. Furthermore, the structure is the same as any official OpenStack charm, of course without functional tests, which is something we cannot provide. Speaking about the deployment process, apart from having a basic charmed OpenStack deployment, operator has to change two tiny configuration options on Nova cloud controller and Cinder. As explained in the Watcher configuration guide, special care has to be done with Oslo notifications for Nova and Cinder [2]. In order to achieve that in charmed OpenStack some issues were met and solved with the following patches [3], [4], [5], [6]. With these patches, operator can set the extra Oslo configuration and this is the only extra configuration needs to take place. Finally, with [7] Keystone charm can accept a relation with Watcher charm instead of ignoring it. To be able to deploy GRNET Watcher charm on Train, patches [3], [4], [5] and [7] have to be back-ported to stable/19.10 branch but that will require the approval of charmers team. Please let me know if such an option is available and in that case I am going to open the relevant patches. Furthermore, if you think that it could be a good option to create a spec and then introduce an official Watcher charm, I would love to help on that. I wish all a happy new year and I am looking forward to your response and possible feedback. PS. If we could have an Ubuntu package for watcher-dashboard [8] like octavia-dashboard [9] we would release a charm for it as well. Best regards, Stamatis Katsaounis [1] https://github.com/grnet/charm-watcher [2] https://docs.openstack.org/watcher/latest/configuration/configuring.html#co… [3] https://review.opendev.org/#/c/699079/ [4] https://review.opendev.org/#/c/699081/ [5] https://review.opendev.org/#/c/699657/ [6] https://github.com/juju/charm-helpers/pull/405 [7] https://review.opendev.org/#/c/699082/ [8] https://github.com/openstack/watcher-dashboard [9] https://launchpad.net/ubuntu/+source/octavia-dashboard -- [cid:part10.101270FA.EDBB3A4A@admin.grnet.gr] Stamatis Katsaounis DevOps Engineer t :(+30) 210 7471130 (ext. 483) f : + 30 210 7474490 GRNET | Networking Research and Education www.grnet.gr<http://www.grnet.gr> | 7, Kifisias Av., 115 23, Athens [Facebook icon]<https://www.facebook.com/grnet.gr/> [Twitter icon] <https://twitter.com/grnet_gr> [Youtube icon] <https://www.youtube.com/user/EDETvideos> [LinkedIn icon] <https://www.linkedin.com/company/55267/> [Instagram icon] <https://www.instagram.com/grnet.gr/>

1 0

[neutron][ironic] dynamic routing protocol status for routed network
by 양유석 08 Jan '20

08 Jan '20

Hi, For ironic flat provider network, I found it's hard to scale manually (for many racks). So I'm trying to use routed network for the purpose. One blur thing for routed network is how we can handle segment connectivity. >From reference (https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html) there is future work about dynamic routing protocol though, I could not find any hints for the functionality. How do you guys using routed network handle the segments' connectivity? Is there any project to speak subnet info via BGP? Thanks in advance.

1 0

VM boot volume disappears from compute's multipath Daemon when NetApp controller is placed offline
by Ahmed ZAKY 07 Jan '20

07 Jan '20

I have a setup where each VM gets assigned two vDisks, one encrypted boot volume and another storage volume. Storage used is NetApp (tripleo-netapp). With two controllers on NetApp side working in active/ active mode. My test case goes as follows: - I stop one of the active controllers. - I stop one of my VMs using OpenStack server stop - I then start my VM one more time using OpenStack server start. - VM fails to start. Here're my findings, hope someone would help if they can explain me the behaviour seen below: My VM: vel1bgw01-MCM2, it is running on compute overcloud-sriovperformancecompute-3.localdomain [root@overcloud-controller-0 (vel1asbc01) cbis-admin]# openstack server show vel1bgw01-MCM2 +--------------------------------------+------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------------------------+------------------------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | zone1 | | OS-EXT-SRV-ATTR:host | overcloud-sriovperformancecompute-3.localdomain | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-sriovperformancecompute-3.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-00000 <+4400000>e93 | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2019-12-18T15:49:37.000000 <+4437000000> | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | SBC01_MGW01_TIPC=192.168.48.22 <+441921684822>; SBC01_MGW01_DATAPATH_MATE=192.168.16.11 <+441921681611>; SBC01_MGW01_DATAPATH=192.168.32.8 <+44192168328> | | config_drive | True | | created | 2019-12-18T15:49:16Z | | flavor | SBC_MCM (asbc_mcm) | | hostId | 7886 <+447886>df0f7a3d4e131304 <+44131304>a8eb860e6a704c5fda2a7ed751b544ff2bf5 | | id | 5c70a984-89 <+4498489>a9-44ce-876d-9e2e568eb819 | | image | | | key_name | CBAM-b5fd59a066e8450 <+448450>ca9f104a69da5a043-Keypair | | name | vel1bgw01-MCM2 | | os-extended-volumes:volumes_attached | [{u'id': u'717e5744-4786-42 <+445744478642>dc-9e3e-3c5e6994 <+446994>c482'}, {u'id': u'd6cf0cf9-36d1-4b 62-86 <+446286>b4-faa4a6642166 <+446642166>'}] | | progress | 0 | | project_id | 41777 <+4441777>c6f1e7b4f8d8fd76b5e0f67e5e8 | | properties | | | security_groups | [{u'name': u'vel1bgw01-TIPC-Security-Group'}] | | status | ACTIVE | | updated | 2020-01-07T17:18:32Z | | user_id | be13deba85794016 <+4485794016>a00fec9d18c5d7cf | +--------------------------------------+------------------------------------------------------------------------------------------------------------+ *It is mapped to the following vDisks (seen using virsh list on compute-3)*: - dm-uuid-mpath-3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714 <+440714> è Boot Volume - dm-uuid-mpath-3600 <+443600>a098000 <+44098000>d9818 <+449818>b 000018565 <+44000018565>dfa069e è Storage volume <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/disk/by-id/dm-uuid-mpath-3600 <+443600>a098000 <+44098000> d9818 <+449818>b0000185 <+440000185>c5dfa0714 <+440714>'/> <backingStore/> <target dev='vda' bus='virtio'/> <serial>717e5744-4786-42 <+445744478642>dc-9e3e-3c5e6994 <+446994> c482</serial> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000 <+440000>' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/disk/by-id/dm-uuid-mpath-3600 <+443600>a098000 <+44098000> d9818 <+449818>b000018565 <+44000018565>dfa069e'/> <backingStore/> <target dev='vdb' bus='virtio'/> <serial>d6cf0cf9-36d1-4b62-86 <+446286>b4-faa4a6642166 <+446642166></serial> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000 <+440000>' bus='0x00' slot='0x08' function='0x0'/> </disk> Name: crypt-dm-uuid-mpath-3600 <+443600>a098000 <+44098000>d9818 <+449818>b 0000185 <+440000185>c5dfa0714 <+440714> State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 5 Number of targets: 1 UUID: CRYPT-LUKS1-769 <+441769>cc20bc5af469c8c9075 <+449075> a2a6fc4aa0-crypt-dm-uuid-mpath-*3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714 <+440714>* Name: *mpathpy* State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 32 Major, minor: 253, *4* Number of targets: 1 UUID: mpath-*3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714 <+440714>* Name: crypt-dm-uuid-mpath-3600 <+443600>a098000 <+44098000>d9818 <+449818>b 000018565 <+44000018565>dfa069e State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 7 Number of targets: 1 UUID: CRYPT-LUKS1-4015 <+4414015>c585a0df4074821 <+444074821> ca312c4caacca-crypt-dm-uuid-mpath-*3600 <+443600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e* Name: *mpathpz* State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 28 Major, minor: 253, *6* Number of targets: 1 UUID: mpath-*3600 <+443600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e* This means boot volume is represented by dm-4 while storage volume is represented by dm-6 Dumping the multipath daemon on the controller shows that at a steady running state both DMs are accounted for (see below). multipathd> show maps name sysfs uuid *mpathpy dm-4 3600 <+4443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714 <+440714>* *mpathpz dm-6 3600 <+4463600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e* mpathqi dm-12 3600 <+44123600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>df5dfafd40 mpathqj dm-13 3600 <+44133600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>de5dfafd10 mpathpw dm-0 3600 <+4403600>a098000 <+44098000>d9818 <+449818>b000018425 <+44000018425>dfa059f mpathpx dm-1 3600 <+4413600>a098000 <+44098000>d9818 <+449818>b0000184 <+440000184>c5dfa05fc mpathqk dm-16 3600 <+44163600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>eb5dfafe80 mpathql dm-17 3600 <+44173600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>e95dfafe26 mpathqh dm-9 3600 <+4493600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>c65dfafa91 These vDisks are mapped to the following multipaths: multipathd> show topology mpathpy (3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714) <+440714> dm-4 NETAPP ,INF-01-00 <+440100> size=21G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw |-+- policy='service-time 0' prio=14 status=active | |- 30:0:0:82 sdm 8:192 active ready running | `- 32:0:0:82 sdk 8:160 active ready running `-+- policy='service-time 0' prio=0 status=enabled |- 33:0:0:82 sdn 8:208 failed faulty running `- 31:0:0:82 sdl 8:176 failed faulty running mpathpz (3600 <+443600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e) dm-6 NETAPP ,INF-01-00 <+440100> size=10G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw |-+- policy='service-time 0' prio=14 status=active | |- 30:0:0:229 sdr 65:16 active ready running | `- 32:0:0:229 sdp 8:240 active ready running `-+- policy='service-time 0' prio=0 status=enabled |- 31:0:0:229 sdo 8:224 failed faulty running `- 33:0:0:229 sdq 65:0 failed faulty running Now, it starts getting very interesting, if I shutdown controller-A from NetApp side, dm-4 disappears but dm-6 is still running while detecting the active path is controller B while standby path is controller-A which now is displayed as failed multipathd> show topology mpathpz (3600 <+443600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e) dm-6 NETAPP ,INF-01-00 <+440100> size=10G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw |-+- policy='service-time 0' prio=0 status=enabled | |- 30:0:0:229 sdr 65:16 failed faulty running | `- 32:0:0:229 sdp 8:240 failed faulty running `-+- policy='service-time 0' prio=11 status=active |- 31:0:0:229 sdo 8:224 active ready running `- 33:0:0:229 sdq 65:0 active ready running multipathd> show maps name sysfs uuid *mpathpz dm-6 3600 <+4463600>a098000 <+44098000>d9818 <+449818>b000018565 <+44000018565>dfa069e* mpathqi dm-12 3600 <+44123600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>df5dfafd40 mpathqj dm-13 3600 <+44133600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>de5dfafd10 mpathpw dm-0 3600 <+4403600>a098000 <+44098000>d9818 <+449818>b000018425 <+44000018425>dfa059f mpathpx dm-1 3600 <+4413600>a098000 <+44098000>d9818 <+449818>b0000184 <+440000184>c5dfa05fc mpathqk dm-16 3600 <+44163600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>eb5dfafe80 mpathql dm-17 3600 <+44173600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>e95dfafe26 mpathqg dm-8 3600 <+4483600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>c75dfafac0 mpathqh dm-9 3600 <+4493600>a098000 <+44098000>d9818 <+449818>b000018 <+44000018>c65dfafa91 If I restore Controller-A into service from NetApp side while fail only the path to controller A from multipathd everything works fine, dm-4 is still present and the VM can be put into service. multipathd> fail path sdk ok multipathd> multipathd> fail path sdm ok mpathpy (3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714) <+440714> dm-4 NETAPP ,INF-01-00 <+440100> size=21G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw |-+- policy='service-time 0' prio=0 status=enabled | |- 32:0:0:82 sdk 8:160 failed faulty running | `- 30:0:0:82 sdm 8:192 failed faulty running `-+- policy='service-time 0' prio=9 status=active |- 31:0:0:82 sdl 8:176 active ready running `- 33:0:0:82 sdn 8:208 active ready running multipathd> reinstate path sdk ok multipathd> multipathd> reinstate path sdm ok mpathpy (3600 <+443600>a098000 <+44098000>d9818 <+449818>b0000185 <+440000185>c5dfa0714) <+440714> dm-4 NETAPP ,INF-01-00 <+440100> size=21G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw |-+- policy='service-time 0' prio=14 status=active | |- 32:0:0:82 sdk 8:160 active ready running | `- 30:0:0:82 sdm 8:192 active ready running `-+- policy='service-time 0' prio=9 status=enabled |- 31:0:0:82 sdl 8:176 active ready running `- 33:0:0:82 sdn 8:208 active ready running It is observed in the working case, the storage volume disappears (which seems normal), also the instance totally vanishes from the virsh list and no trace can be found at the KVM level if we run ps -def | grep fd | grep <instance_ID>. However, the boot volume is always present in the multipathd records when we stop the VM at normal conditions without stopping NetApp controller. Any ideas? Kind regards, Ahmed

1 0

[infra] Retire x/dox
by Andreas Jaeger 07 Jan '20

07 Jan '20

The x/dox repo is unused, let's retire it. I'll put up changes with topic "retire-dox", Andreas -- Andreas Jaeger aj(a)suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB

1 0

[cinder] first meeting of 2020 tomorrow (8 January)
by Brian Rosmaita 07 Jan '20

07 Jan '20

Just wanted to send a quick reminder that the first Cinder team meeting of 2020 well be held tomorrow (8 January) on the usual day (Wednesday) at the usual time (1400 UTC) and in the usual place (#openstack-meeting-4). https://etherpad.openstack.org/p/cinder-ussuri-meetings See you there! brian

1 0

IBM z/VM CI is planning to migrate new environment
by Dong JV Ma 07 Jan '20

07 Jan '20

Hello, Due to the hardware issues, the IBM z/VM CI system [1] is planning to migrate to the new hardware in the next few months, the planning time will be in 3 months. We will do following changes with IBM z/VM CI environment: 1. Migrate the IBM z/VM CI to new hardware 2. Upgrade IBM z/VM CI from python2 to python3 During this time, IBM z/VM CI will not stable, please ignore the IBM z/VM CI failures. For any questions feel free to reach out to me(Dong Ma). [1] https://wiki.openstack.org/wiki/ThirdPartySystems/IBM_z/VM_CI Thanks & Best Regards! Dong Ma (马冬) LinuxONE IaaS Infra & CI/CD, China Systems Lab E-mail: madongma(a)cn.ibm.com IRC: larainema Address: 2/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC

3 2

[tc] Expediting patch to unblock governance's CI
by Jean-Philippe Evrard 07 Jan '20

07 Jan '20

Hello everyone, Our governance repo's testing is currently failing, and I have patches to fix it (short term [1] and long term). Because it's been a while it's in there (and got recently updated), I will now merge the short term fix, even if we don't have enough votes (nor time). It will unblock many patches and prevent useless rechecks, at the expense of being out of policy. I will gladly take the blame, should there be any. Regards, JP [1] https://review.opendev.org/#/c/700422/

1 0

[kolla] neutron-l3-agent namespace NAT table not working?
by Jon Masters 06 Jan '20

06 Jan '20

Hi there, I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the iptables rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or SNAT applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING chains (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log entries. It's as if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is driving me crazy :) Anyone got some quick suggestions? (assume I tried the obvious stuff). Jon. -- Computer Architect

8 19

[Scientific] Scientific SIG meeting July 7th 2100 UTC
by Martial Michel 06 Jan '20

06 Jan '20

The first meeting of the new year at 2100 UTC on Tuesday, July 7th. Mostly an Any Other Business meeting to get its participants back from the holidays as reflected by the agenda :) https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_January_7th_2020

1 0

[all] Is there something I can do to get a simple fix done?
by Neil Jerram 06 Jan '20

06 Jan '20

I'm struggling to say this positively, but... it feels like OpenStack promotes refactoring work that will likely break something, but is very slow when a corresponding fix is needed, even when the fix is trivial. Is there something we could do to get fixes done more quickly when needed? My case in point: my team's networking plugin (networking-calico) does not do "extraroutes", and so was broken by some python-openstackclient change (possibly [1]) that wrongly assumed that. I posted a fix [2] that passed CI on 21st October, and asked for it to be reviewed on IRC a couple of days later. Édouard Thuleau posted a similar fix [3] on 4th December, and we agreed that his was better, so I abandoned mine. His fix attracted a +2 on 11th December, but has been sitting like that ever since. It's a fix that I would expect to be simple to review, so I wonder if there's something else we could have done here to get this moving? Or if there is a systemic problem here that deserves discussion? [1] https://opendev.org/openstack/python-openstackclient/commit/c44f26eb7e41c28… [2] https://review.opendev.org/#/c/685312/ [3] https://review.opendev.org/#/c/697240/ Many thanks, Neil

2 1

[all][neutron][neutron-vpnaas] Maintainers needed
by Slawek Kaplonski 06 Jan '20

06 Jan '20

Hi, Over the past couple of cycles we have noticed that new contributions and maintenance efforts for neutron-vpnaas were almost non existent. This impacts patches for bug fixes, new features and reviews. The Neutron core team is trying to at least keep the CI of this project healthy, but we don’t have enough knowledge about the details of the neutron-vpnaas code base to review more complex patches. During the PTG in Shanghai we discussed that with operators and TC members during the forum session [1] and later within the Neutron team during the PTG session [2]. During these discussions, with the help of operators and TC members, we reached the conclusion that we need to have someone responsible for maintaining project. This doesn’t mean that the maintainer needs to spend full time working on this project. Rather, we need someone to be the contact person for the project, who takes care of the project’s CI and review patches. Of course that’s only a minimal requirement. If the new maintainer works on new features for the project, it’s even better :) If we don’t have any new maintainer(s) before milestone Ussuri-2, which is Feb 10 - Feb 14 according to [3], we will need to mark neutron-vpnaas as deprecated and in “V” cycle we will propose to move the project from the Neutron stadium, hosted in the “openstack/“ namespace, to the unofficial projects hosted in the “x/“ namespace. So if You are using this project now, or if You have customers who are using it, please consider the possibility of maintaining it. Otherwise, please be aware that it is highly possible that the project will be deprecated and moved out from the official OpenStack projects. [1] https://etherpad.openstack.org/p/PVG-Neutron-stadium-projects-the-path-forw… [2] https://etherpad.openstack.org/p/Shanghai-Neutron-Planning-restored - Lines 379-421 [3] https://releases.openstack.org/ussuri/schedule.html -- Slawek Kaplonski Senior software engineer Red Hat

1 1

[neutron] Bug deputy report - week of 30th December
by Slawek Kaplonski 06 Jan '20

06 Jan '20

Hi, I was on bug deputy last week. It was pretty quiet week with only few bugs reported. Below is my summary of it. Critical: https://bugs.launchpad.net/neutron/+bug/1858260 - Upstream CI neutron-tempest-plugin-* fails - I marked it as critical as it cause gate failures, I will have to take a look at it closer next week, Medium: https://bugs.launchpad.net/neutron/+bug/1858086 - qrouter's local link route cannot be restored - confirmed by me on local env, it would be good if someone from L3 subteam can take a look at it, Undecided and others: https://bugs.launchpad.net/neutron/+bug/1858377 - probably bug for openstackclient rather than neutron, but I would like to wait for confirmation from bug reporter first, https://bugs.launchpad.net/neutron/+bug/1858262 - duplicate of other bug, https://bugs.launchpad.net/neutron/+bug/1858419 - docs bug to update config options for large scale deployments, see http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011820.… for details — Slawek Kaplonski Senior software engineer Red Hat

1 0

[oslo][kolla][requirements][release][infra] Hit by an old, fixed bug
by Radosław Piliszek 06 Jan '20

06 Jan '20

Hi Folks, as the subject goes, my installation has been hit by an old bug: https://bugs.launchpad.net/oslo.messaging/+bug/1828841 (bug details not important, linked here for background) I am using Stein, deployed with recent Kolla-built source-based images (with only slight modifications compared to vanilla ones). Kolla's procedure for building source-based images considers upper constraints, which, unfortunately, turned out to be lagging behind a few releases w.r.t. oslo.messaging at least. The fix was in 9.7.0 released on May 21, u-c still point to 9.5.0 from Feb 26 and the latest of Stein is 9.8.0 from Jul 18. It seems oslo.messaging is missing from the automatic updates that bot proposes: https://review.opendev.org/#/q/owner:%22OpenStack+Proposal+Bot%22+project:o… Per: https://opendev.org/openstack/releases/src/branch/master/doc/source/referen… this upper-constraint proposal should be happening for all releases. I would be glad if someone investigated why it happens(/ed) and audited whether other OpenStack projects don't need updating as well to avoid running on old deps when new are awaiting for months. :-) Please note this might apply to other branches as well. PS: for some reason oslo.messaging Stein release notes ( https://docs.openstack.org/releasenotes/oslo.messaging/stein.html ) are stuck at 9.5.0 as well, this could be right (I did not inspect the sources) but I am adding this in PS so you have more things to correlate if they need be. -yoctozepto

5 7

[kolla] Adding Dincer Celik to kolla-core and kolla-ansible-core
by Mark Goddard 06 Jan '20

06 Jan '20

Hi, I recently proposed to the existing cores that we add Dincer Celik (osmanlicilegi) to the kolla-core and kolla-ansible-core groups and we agreed to go ahead. Thanks for your contribution to the project so far Dincer, I'm glad to have you on the team. Cheers, Mark

1 0

[neutron][nova][cinder][glance][largescale-sig] Documentation update for large-scale
by Arnaud Morin 06 Jan '20

06 Jan '20

Hey all, With the new "Large scale SIG", we were thinking about updating documentation to help operators setting up large deployments. To do so, we would like to propose, at least, some documentation changes to identify options that affect large scale. The plan is to have a small note on some options and eventually a link to a specific page for large scale (see attachments). I know that nova started working on collecting those parameters here: https://bugs.launchpad.net/nova/+bug/1838819 Do you know if something similar exists on other projects? Moreover, we would like to collect more parameters that could be tuned on a large scale deployment, for every project. So, if you have any, feel free to answer to this mail or add some info on the following etherpad: https://etherpad.openstack.org/p/large-scale-sig-documentation Thanks for your help! Regards. The large-scale team. -- Arnaud Morin

2 1

[cloudkitty] Stepping down from PTL
by Luka Peschke 06 Jan '20

06 Jan '20

Hi all, I'm moving to a new position that doesn't involve OpenStack, and won't leave me the required time to be Cloudkitty's PTL. This is why I have to step down from the PTL position. jferrieu will take my position for the end of the U cycle (he's been a major contributor recently), with the help of huats, who's been the Cloudkitty PTL before me, and has been around in the community for a long time. I've been the PTL for two and a half cycles, and I think that it is a good thing for the project to take a new lead, with a new vision. I'm grateful for my experience within the OpenStack community. Thanks, -- Luka Peschke (peschk_l)

3 2

[auto-scaling][self-healing] Discussion to merge two SIG to one
by Rico Lin 05 Jan '20

05 Jan '20

Dear all As we discussed in PTG about merge two SIG to one. I would like to continue the discussion on ML. In PTG, Eric proposes the idea to merge two SIG due to the high overlapping of domains and tasks. I think this is a great idea since, over the last 6 months, most of the discussions in both SIG are overlapped. So I'm onboard with this idea. Here's how I think we can continue this idea: 1. Create new SIG (maybe 'Automation SIG'? feel free to propose name which can cover both interest.) 2. Redirect docs and wiki to new SIG. And rework on index so there will be no confusion 3. Move repos from both SIGs to new SIG 4. Mark auto-scaling SIG and self-healing SIG as inactive. 5. remove auto-scaling SIG and self-healing SIG after a reasonable waiting time Let us know what you think about this. Otherwise, we definitely expect this to happen soon. -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

4 7

[meta-sig][multi-arch] propose forming a Multi-arch SIG
by Rico Lin 05 Jan '20

05 Jan '20

Dear all In summit, there's a forum for ARM support [1] in Summit which many people show they're interested in ARM support for OpenStack. And since also we have Linaro shows interest in donating servers to OpenStack infra. It's time for community to think about what we should deal with those ARM servers once we have them in community infrastructure. One thing we should do as a community is to gather people for this topic. So I propose we create a Multi-arch SIG and aim to support ARM architecture as very first step. I had the idea to call it ARM SIG before, but since there might be high overlap knowledge between support ARM 64 and other architectures. I propose we go for Multi-arch instead. This SIG will be a nice place to collect all the documents, gate jobs, and to trace tasks. If you're also interested in that group, please reply to this email, introduce yourself and tell us what you would like the group scope and objectives to be, and what you can contribute to the group. [1] https://www.openstack.org/summit/shanghai-2019/summit-schedule/events/24355… -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

11 15

About the use of security groups with neutron ports
by ahmed.zaky.abdallah＠gmail.com 04 Jan '20

04 Jan '20

Hi All, I am trying to wrap my head around something I came across in one of the OpenStack deployments. I am running Telco VNFs one of them is having different VMs using SR-IOV interfaces. On one of my VNFs on Openstack, I defined a wrong IPv6 Gm bearer interface to be exactly the same as the IPv6 Gateway. As I hate re-onboarding, I decided to embark on a journey of changing the IPv6 of the Gm bearer interface manually on the application side, everything went on fine. After two weeks, my customer started complaining about one way RTP flow. The customer was reluctant to blame the operation I carried out because everything worked smooth after my modification. After days of investigation, I remembered that I have port-security enabled and this means AAP “Allowed-Address-Pairs” are defined per vPort (AAP contain the floating IP address of the VM so that the security to allow traffic to and from this VIP). I gave it a try and edited AAP “Allowed-Address-Pairs” to include the correct new IPv6 address. Doing that everything started working fine. The only logical explanation at that time is security group rules are really invoked. Now, I am trying to understand how the iptables are really invoked. I did some digging and it seems like we can control the firewall drivers on two levels: * Nova compute * ML2 plugin I was curious to check nova.conf and it has already the following line: firewall_driver=nova.virt.firewall.NoopFirewallDriver However, checking the ml2 plugin configuration, the following is found: 230 [securitygroup] 231 232 # 233 # From neutron.ml2 234 # 235 236 # Driver for security groups firewall in the L2 agent (string value) 237 #firewall_driver = <None> 238 firewall_driver = openvswitch So, I am jumping to a conclusion that ml2 plugin is the one responsible for enforcing the firewall rules in my case. Have you had a similar experience? Is my assumption correct: If I comment out the ml2 plugin firewall driver then the port security carries no sense at all and security groups won’t be invoked? Cheers, Ahmed

4 4

[stein][cinder][backup][swift] issue
by Ignazio Cassano 03 Jan '20

03 Jan '20

Hello All, I configured openstack stein on centos 7 witch ceph. Cinder works fine and object storage on ceph seems to work fine: I can clreate containers, volume etc ..... I configured cinder backup on swift (but swift is using ceph rados gateway) : backup_driver = cinder.backup.drivers.swift.SwiftBackupDriver swift_catalog_info = object-store:swift:publicURL backup_swift_enable_progress_timer = True #backup_swift_url = http://10.102.184.190:8080/v1/AUTH_ backup_swift_auth_url = http://10.102.184.190:5000/v3 backup_swift_auth = per_user backup_swift_auth_version = 1 backup_swift_user = admin backup_swift_user_domain = default #backup_swift_key = <None> #backup_swift_container = volumebackups backup_swift_object_size = 52428800 #backup_swift_project = <None> #backup_swift_project_domain = <None> backup_swift_retry_attempts = 3 backup_swift_retry_backoff = 2 backup_compression_algorithm = zlib If I run a backup as user admin, it creates a container named "volumebackups". If I run a backup as user demo and I do not specify a container name, it tires to write on volumebackups and gives some errors: ClientException: Container PUT failed: http://10.102.184.190:8080/swift/v1/AUTH_964f343cf5164028a803db91488bdb01/v… 409 Conflict BucketAlreadyExists Does it mean I cannot use the same containers name on differents projects ? My ceph.conf is configured for using keystone: [client.rgw.tst2-osctrl01] rgw_frontends = "civetweb port=10.102.184.190:8080" # Keystone information rgw keystone api version = 3 rgw keystone url = http://10.102.184.190:5000 rgw keystone admin user = admin rgw keystone admin password = password rgw keystone admin domain = default rgw keystone admin project = admin rgw swift account in url = true rgw keystone implicit tenants = true Any help, please ? Best Regards Ignazio

3 4

[openstack-ansible] strange execution delays
by Joe Topjian 02 Jan '20

02 Jan '20

Hello, I don't think this is an OSA issue per-se, but it's happening on infrastructure deployed with OSA. Using both the Rocky and Stein releases of OSA on Ubuntu 18.04, we're running into a strange a strange issue where the run times of Ansible will eventually take longer and longer. Digging into the issue more, we find that it's when Ansible executes a command within a container and the call to "su" triggers the delay. Over time, the delay increases more and more. The same behavior is seen if we SSH into containers. The only workaround we've been able to identify that works is to stop and start the containers. We have not been able to identify a root cause of this, nor any consistency to what containers this will happen on. Has anyone run into this? Thanks, Joe

2 5

Re: openstack-discuss Digest, Vol 15, Issue 1
by Soniya Vyas 02 Jan '20

02 Jan '20

> Message: 4 > Date: Tue, 31 Dec 2019 19:03:22 -0600 > From: Ghanshyam Mann <gmann(a)ghanshyammann.com> > To: "openstack-discuss" <openstack-discuss(a)lists.openstack.org> > Subject: [qa] QA Office hour new timing for 2020 > Message-ID: > <16f5ea0ef6f.1010065b175496.3744470588375359656(a)ghanshyammann.com> > Content-Type: text/plain; charset="UTF-8" > > Hello Everyone, > > We have few members who actively started contribution to QA from India as well as from Europe TZ. > The current office hour time is more convenient for CT and JST only and very difficult for members from India > Europe to join. > > I would like to adjust office hour timing to include all four TZs. To do that someone has to wake up early or stay > late at night :). I gave preference to new members and selected Tuesday 13.30 UTC [1]. which will be an early morning > for me and late-night in Tokyo. > > let me know if any objection or better suggestion. Accordingly, I will make the new time effective from 7th Jan. > > Also cancelling the 1st Jan office hour and wish you all a very happy new year. > > [1] https://www.timeanddate.com/worldclock/meetingdetails.html?year=2020&month=… Thanks a lot to whole QA team and Ghanshyam Mann for considering our timing issues. Looking forward to join QA Office hours meeting. Regards, Soniya Vyas

1 0

[manila] No IRC/Community meeting on 2nd January 2020
by Goutham Pacha Ravi 01 Jan '20

01 Jan '20

Hello Zorillas, Due to the holidays we're expecting multiple community members (myself included) to be unavailable for the weekly IRC meeting tomorrow (2nd January 2020, 15:00 UTC), so we'll skip it. Please add any agenda items to the next meeting (9th January 2020, 15:00 UTC) [1]. Happy New Year, and the rest of your holidays! Thanks, Goutham [1] https://wiki.openstack.org/wiki/Manila/Meetings

1 0

[qa] QA Office hour new timing for 2020
by Ghanshyam Mann 31 Dec '19

31 Dec '19

Hello Everyone, We have few members who actively started contribution to QA from India as well as from Europe TZ. The current office hour time is more convenient for CT and JST only and very difficult for members from India Europe to join. I would like to adjust office hour timing to include all four TZs. To do that someone has to wake up early or stay late at night :). I gave preference to new members and selected Tuesday 13.30 UTC [1]. which will be an early morning for me and late-night in Tokyo. let me know if any objection or better suggestion. Accordingly, I will make the new time effective from 7th Jan. Also cancelling the 1st Jan office hour and wish you all a very happy new year. [1] https://www.timeanddate.com/worldclock/meetingdetails.html?year=2020&month=… -gmann

1 0

[neutron] Slow floating IP activation when FWaaS v2 enabled
by Jason Anderson 31 Dec '19

31 Dec '19

Hi all, We recently started using the FWaaS feature (v2) in Neutron for our deployment. However, we noticed that when this feature is enabled, floating IP activation (the setting up of iptables rules that perform DNAT to the destination instance) takes a long time to happen. The effect is that the user has to wait for a while before the floating IP becomes active on an instance. I was wondering if anybody else has experienced this, and if there are any workarounds. I suspect there may be some queuing happening behind an iptables lock somewhere but I haven't been able to verify anything for sure yet. Thank you, -- Jason Anderson Chameleon DevOps Lead Consortium for Advanced Science and Engineering, The University of Chicago Mathematics & Computer Science Division, Argonne National Laboratory

1 0

[horizon][requirements] django-3 blocked by usage of django-babel
by Matthew Thode 31 Dec '19

31 Dec '19

it looks like django-babel is unmaintained since 2017, so perhaps it's time to look for a replacement. Looking through github issues, it looks like enmerkar is a possible replacement. If that's fine with horizon / django consumers we can start by adding enmerkar to global-requirements and upper-constraints. Thanks, -- Matthew Thode

4 9

[neutron] Bug deputy report week of December 23rd
by Miguel Lavalle 30 Dec '19

30 Dec '19

Hi, I was the bug deputy for the week of December 23rd. As expected for an end of year Holidays week, We had very little activity: Requested more information: - https://bugs.launchpad.net/neutron/+bug/1857422 neutron-keepalived-state-change and keeplived cannot be cleanup for those routers which is deleted during l3-agent died

2 1

Openstack Rocky to Train upgrade
by Md. Farhad Hasan Khan 30 Dec '19

30 Dec '19

Hi, Could you please help me how can I upgrade from Openstack Rocky version to Train. Thanks & B'Rgds, Rony

3 5

[devstack][ceilometer][tacker]ceilometer devstack installation fails
by Hiroyuki JO 25 Dec '19

25 Dec '19

Hi ceilometer team and all, After a patch[1] was merged, Tacker zuul check fails because ceilometer devstack installation error[2]. Do you have any work aroud of this problem? Addition, why devstack uses pip version 9.0.3 when installing python packages? The cause of our problem is installation error of conluent-kafka. But with latest pip (version 19), I was able to install the package. [1] https://opendev.org/openstack/requirements/commit/c99ab09bc44818b92fe86842f… [2] https://zuul.opendev.org/t/openstack/build/e6d04b35d16e4447901aba441721115a… Regards, 徐広幸 (Hiroyuki JO) Email: hiroyuki.jo.mt(a)hco.ntt.co.jp / Tel(direct): +81-422-59-7394 NTT Network Systems Laboratories

3 5

[cyborg] No weekly IRC meeting for next 2 weeks
by Nadathur, Sundar 23 Dec '19

23 Dec '19

There will be no weekly Cyborg IRC meeting on Dec 26 or Jan 2. Happy Holidays to all. Regards, Sundar

1 0

[neutron] bug deputy report of week 2019-12-16
by Bence Romsics 23 Dec '19

23 Dec '19

Hi Neutron Team, Here comes the deputy report of last week: High: * https://bugs.launchpad.net/neutron/+bug/1856600 Unit test jobs are failing with ImportError: cannot import name 'engine' from 'flake8' Gate unblocking fix by Brian merged already. Follow up proposed by Nate: https://review.opendev.org/699243 Medium: * https://bugs.launchpad.net/neutron/+bug/1856523 Sometimes instance can't get public keys due to cirros metadata request failur Occasional loss of metadata response. Follow up bug report for CirrOS by Maciej: https://github.com/cirros-dev/cirros/issues/8 and pull request on github: https://github.com/cirros-dev/cirros/pull/11 * https://bugs.launchpad.net/neutron/+bug/1856675 / https://bugs.launchpad.net/neutron/+bug/1856726 IPv6 Prefix Delegation do not work / IpamBackendMixin._gateway_ip_str is not reading correctly the subnet IP version Fix proposed by Rodolfo: https://review.opendev.org/699465 * https://bugs.launchpad.net/neutron/+bug/1856853 OVSInterfaceDriver.plug_new should catch pyroute2.netlink.exceptions.NetlinkErro Fix proposed by Rodoflo: https://review.opendev.org/699700 * https://bugs.launchpad.net/neutron/+bug/1857021 Wrong order of adding/deleting router ports may end up with not configured router port Fix proposed by Slawek: https://review.opendev.org/700011 Low: * https://bugs.launchpad.net/neutron/+bug/1857035 Add better description for "test_cleanup_network_namespaces_cleans_dhcp_and_l3_namespaces" in case of failure Low hanging fruit improvement to logging. Unassigned. * https://bugs.launchpad.net/neutron/+bug/1857047 openstack port set --no-fixed-ips removed only 1 fixed IP Unassigned. Undecided: * https://bugs.launchpad.net/neutron/+bug/1857016 Possible double row.delete() call in ACL code neutron fix by Terry in the gate: https://review.opendev.org/700007 the same change in neutron-ovn: https://review.opendev.org/700001 Maybe RFE: * https://bugs.launchpad.net/neutron/+bug/1856839 [L3] router processing time increase if there are large large set ports Possbile performance problem reported by Yulong (who is also the assignee). No gerrit change yet. Merry Christmas, Bence

1 0

[heat] Addressing the large patch backlog.
by David Peacock 23 Dec '19

23 Dec '19

Hi all, For those interested in the workings of the Heat project, I'd like to kick off a call to action. At the time of writing there are approximately 200 open patches against the core Heat project repo alone, not counting the other Heat repos. Recently I started going through and triaging the patches I'd consider "historical" with an arbitrary cut off for this definition of August 1st of this year. There are 148 patches which meet this definition, dating all the way back to 2015. I have gone through them all and placed them into a spreadsheet [0] which I'd invite you all to check. Provided is a link to the patch in question, initial upload date, last meaningful update date, primary author, and a high level summary of the patch. Additionally I've broken the patches down into three recommended states based on a high level first pass. *Abandon* 34 patches are candidates to be abandoned; they usually are of debatable utility, have significant outstanding concerns, or have no followup from the original developer in a very long time. In many cases, all of these conditions. *Without good reason or explanation from the original developer, these patches may ultimately be cleared out.* *Rebase + Merge* 38 patches are with a high level look in reasonably good shape, perform a stated goal, and may be trivial to core review and ultimately rebase and merge. *If you're the original developer or otherwise interested in these patches and wish to see them through the merge process, please rebase the patch.* *Research* 76 patches are sufficiently complex that they'll need a much closer look. Some of these patches are in a seemingly "finished" state, some are a way off. Some have unanswered concerns from core review and have been left dangling. *If you're the original developer or otherwise interested in working these patches through to completion, please do get involved.* When I started this little mission I wasn't quite sure what to expect. What I have found is that as much as there was anticipated cruft to clear out, there is a great deal of very good work lurking here, waiting to see the light of day, and it would be so good to see this work realised. :-) If you have anything to say, feel free to write back on list, and if you'd like to coordinate with me any efforts with these patches I can be found by email or on Freenode in the #heat channel; I'm dpeacock. Based on feedback of this idea, and indeed on each individual patch, I hope we can get this backlog under control, and harvest some of this excellent code! Thank you, David Peacock [0] https://ethercalc.openstack.org/b3qtqyhkg9g1 Please be mindful of accidental edits.

2 2

[kuryr] PTL on vacations
by Michał Dulko 23 Dec '19

23 Dec '19

Hi, I'm starting my vacations now and will be back on January 13th. Please direct any issues to the rest of the core team. Thanks, Michał

1 0

[tripleo] TripleO CI Summary: Sprint 40
by Rafael Folco 23 Dec '19

23 Dec '19

Greetings, The TripleO CI team has just completed Sprint 40 / Unified Sprint 19 (Nov 21 thru Dec 18). The following is a summary of completed work during this sprint cycle: - Created a PoC for 3rd-party testing jobs for podman and ceph-ansible. The 3rd-party jobs are now running and being triggered from github pull requests. - Closed-out work on the design enhancements to the promotion server. Tested the new promoter code and fixed issues related to manifests and quay.io. - Working w/ quay support re: performance and api issues. - Implemented component pipeline [3] running daily and promoting separated from the integration jobs. Job that promotes the component still under testing and fixing bugs. Component jobs have been also created in downstream. - Addressed issues and technical debt tasks in TripleO CI realm, including issues in the zuul reproducer. Next sprint starts Jan 3rd. The planned work for the next sprint [1] are still going to be defined, and may include component pipeline work and build out CentOS8 promotion and check jobs upstream. The Ruck and Rover for this sprint are TBD. Please direct questions or queries to them regarding CI status or issues in #tripleo, ideally to whomever has the ‘|ruck’ suffix on their nick. Ruck/rover notes to be tracked in etherpad [2]. Thanks, rfolco [1] https://tree.taiga.io/project/tripleo-ci-board/taskboard/unified-sprint-20 [2] https://etherpad.openstack.org/p/ruckroversprint20 [3] http://dashboard-ci.tripleo.org/d/UDA4H3aZk/component-pipeline?orgId=1

1 0

[kuryr] Multi-net spec 1.1 released
by Michał Dulko 23 Dec '19

23 Dec '19

Hi, I just wanted to give a heads up that version 1.1 of Multi-Network CRD Specification was just released. It might be worth to implement support for it in Kuryr at some point if there's something we could benefit from. Thanks, Michał

1 0

[octavia] Holiday Meeting Cancellations!
by Adam Harwell 21 Dec '19

21 Dec '19

Hey all! Octavia meetings will be cancelled for the next two weeks (Dec 25 and Jan 1), and will resume on Jan 8. See y'all then, and have a great holiday season! --Adam

1 0

[horizon] team meeting cancelled in the holiday season
by Akihiro Motoki 20 Dec '19

20 Dec '19

Hi, Horizon team cancel the team meetings at Dec 25, 2019 and Jan 1, 2020. See you all in new year and enjoy holidays! Thanks, Akihiro Motoki (irc: amotoki)

1 0

State of the Gate
by Matt Riedemann 20 Dec '19

20 Dec '19

Things are great! Surprise! I just wanted to let everyone know. Later! . . . . . Now that you've been tricked, on Halloween no less, I'm here to tell you that things suck right now. This is your periodic digest of issues. Grab some fun-sized candy bars and read on. I think right now we have three major issues. 1. http://status.openstack.org/elastic-recheck/index.html#1763070 This has resurfaced and I'm not sure why, nor do I think we ever had a great handle on what is causing this or how to work around it so if anyone has new ideas please chip in. 2. http://status.openstack.org/elastic-recheck/index.html#1844929 I've done some digging into this one and my notes are in the bug report. It mostly affects grenade jobs but is not entirely restricted to grenade jobs. It's also mostly on OVH and FortNebula nodes but not totally. From looking at the mysql logs in the grenade jobs, mysqld is (re)started three times. I think (1) for initial package install, (2) for stacking devstack on the old side, and (3) for stacking devstack on the new side. After the last restart, there are a lot of aborted connection messages in the msyql error log. It's around then that grenade is running post-upgrade smoke tests to create a server and the nova-scheduler times out communicating with the nova_cell1 database. I have a few patches up to grenade/devstack [1] to try some things and get more msyql logs but so far they aren't really helpful. We need someone with some more mysql debugging experience to help here, maybe zzzeek or mordred? 3. CirrOS guest SSH issues There are several (some might be duplicates): http://status.openstack.org/elastic-recheck/index.html#1848078 http://status.openstack.org/elastic-recheck/index.html#1808010 (most hits) http://status.openstack.org/elastic-recheck/index.html#1463631 http://status.openstack.org/elastic-recheck/index.html#1849857 http://status.openstack.org/elastic-recheck/index.html#1737039 http://status.openstack.org/elastic-recheck/index.html#1840355 http://status.openstack.org/elastic-recheck/index.html#1843610 A few notes here. a) We're still using CirrOS 0.4.0 since Stein: https://review.opendev.org/#/c/521825/ And that image was published nearly 2 years ago and there are no newer versions on the CirrOS download site so we can't try a newer image to see if that fixes things. b) Some of the issues above are related to running out of disk in the guest. I'm not sure what is causing that, but I have posted a devstack patch that is related: https://review.opendev.org/#/c/690991 tl;dr before Stein the tempest flavors we used had disk=0 so nova would fit the root disk to the size of the image. Since Stein the tempest flavors specify root disk size (1GiB for the CirrOS images). My patch pads an extra 1GiB to the root disk on the tempest flavors. One side effect of that is the volumes tempest creates will go from 1GiB to 2GiB which could be a problem if a lot of tempest volume tests run at the same time, though we do have a volume group size of 24GB in gate runs so I think we're OK for now. I'm not sure my patch would help, but it's an idea. As for the other key generation and dhcp lease failures, I don't know what to do about those. We need more eyes on these issues to generate some ideas or see if we're doing something wrong in our tests, e.g. generating too much data for the config drive? Not using config drive in some cases? Metadata API server is too slow (note we cache the metadata since [2])? Other ideas on injecting logs for debug? [1] https://review.opendev.org/#/q/topic:bug/1844929+status:open [2] https://review.opendev.org/#/q/I9082be077b59acd3a39910fa64e29147cb5c2dd7 -- Thanks, Matt

7 23

Is there a limit on the number of VMs returned by "openstack server list --all-projects" ?
by Albert Braden 20 Dec '19

20 Dec '19

2 4

[fuel][fuel-ccp] Retire Fuel and fuel-ccp repositories
by Andreas Jaeger 20 Dec '19

20 Dec '19

The fuel project has been removed from from OpenStack in 2017 and has no action for some time, the fuel-ccp repositories are dead as well. I've talked with Sergey from Mirantis and he confirmed now that those can be retired. I'll take care of retiring these repositories now and will retire all fuel repos in OpenStack namespace as well as the fuel-ccp repositories in the x namespace. I'll use topic retire-fuel for the changes, Andreas P.S. full list of repos: openstack/fuel-agent openstack/fuel-astute openstack/fuel-dev-tools openstack/fuel-devops openstack/fuel-docs openstack/fuel-library openstack/fuel-main openstack/fuel-menu openstack/fuel-mirror openstack/fuel-nailgun-agent openstack/fuel-nailgun-extension-cluster-upgrade openstack/fuel-nailgun-extension-converted-serializers openstack/fuel-nailgun-extension-iac openstack/fuel-octane openstack/fuel-ostf openstack/fuel-plugin-murano openstack/fuel-plugin-murano-tests openstack/fuel-plugins openstack/fuel-qa openstack/fuel-specs openstack/fuel-stats openstack/fuel-ui openstack/fuel-upgrade openstack/fuel-virtualbox openstack/fuel-web openstack/python-fuelclient x/fuel-ccp x/fuel-ccp-ceph x/fuel-ccp-ci-config x/fuel-ccp-cinder x/fuel-ccp-debian-base x/fuel-ccp-designate x/fuel-ccp-elasticsearch x/fuel-ccp-entrypoint x/fuel-ccp-etcd x/fuel-ccp-galera x/fuel-ccp-glance x/fuel-ccp-grafana x/fuel-ccp-heat x/fuel-ccp-horizon x/fuel-ccp-installer x/fuel-ccp-ironic x/fuel-ccp-keystone x/fuel-ccp-mariadb x/fuel-ccp-memcached x/fuel-ccp-murano x/fuel-ccp-neutron x/fuel-ccp-nginx x/fuel-ccp-nova x/fuel-ccp-openstack-base x/fuel-ccp-rabbitmq x/fuel-ccp-rally x/fuel-ccp-sahara x/fuel-ccp-searchlight x/fuel-ccp-specs x/fuel-ccp-stacklight x/fuel-ccp-tests x/fuel-ccp-zmq -- Andreas Jaeger aj(a)suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB

2 9

[kolla] broken images pushed
by Marcin Juszkiewicz 20 Dec '19

20 Dec '19

We have merged publisher for aarch64 images and it turned out that it was done wrong way. Today both x86-64 and aarch64 images got published at same time so "kolla/debian-source-*" images are broken: 08:55 (0s) hrw@puchatek:~$ docker run --rm -it kolla/debian-source-heat-engine:master bash standard_init_linux.go:211: exec user process caused "exec format error" This is result of aarch64 image overwriting x86-64 one. We reverted arm64 publisher and will work on it in January. Sorry for potential problem.

1 1

[Horizon][Designate][Tacker][Tobiko][OSH][Ironic] Flaky jobs and retries
by Clark Boylan 20 Dec '19

20 Dec '19

Hello, Zuulv3 has a feature where it will rerun jobs that it thinks failed due to infrastructure problems up to some configured limit. For us the limit is 3 job attempts. The circumstances where Zuul thinks it has infrastructure issues are when pre-run playbooks fail (because these are expected to be stable), and when Ansible returns with an exit code signifying network connectivity issues. Recently I updated Zuul to report the job attempt in job inventories and am using that to report this data into logstash and elasticsearch. Now I can use queries like: 'zuul_attempts:"3" AND filename:"job-output.txt" AND message:"Job console starting"' to see which jobs are flaky and reattempting. I'm writing this email to report on some of what I've found in the hopes that the problems we have can be fixed and hopefully result in better jobs across the board. Designate Horizon Dashboard The nodejs jobs for stable/rocky expect to run against nodejs4 but run on Bionic which has no nodejs 4 packaging. This causes the pre-run to fail as it cannot install the requested package, https://zuul.opendev.org/t/openstack/build/498ebb052e2b4a3393b0939820ee8927…. Fix in progress here, https://review.opendev.org/#/c/699241/1, to pin to Xenial. Thank you! Horizon projects and dashboards may want to double check they don't have similar problems with nodejs version and node mismatches. Tacker tacker-functional-devstack-multinode-python3 (and possibly other tacker devstack jobs) attempt to download an openwrt image during devstack runs and the server isn't responding. This fails in pre-run, http://zuul.openstack.org/build/27ff514c77724250968d60469923f613/log/contro…, and results in the job being retried. I am not aware of a fix for this, but you'll likely need to find a new image host. Let the infra team know if we can help. Tobiko tobiko-devstack-faults-centos-7 is failing because this job runs on CentOS 7 using python2 but Nova needs python3.6 now. This fails in pre-run, https://zuul.opendev.org/t/openstack/build/8071780e7ba748169b447f1d42e069fc…, and forces the job to be rerun. I'm not sure what the best fix is here. Maybe run your jobs on Ubuntu until CentOS 8 is supported by devstack? OpenStack Helm Ansible has decided that become_user is not valid on include_role tasks. https://opendev.org/openstack/openstack-helm-infra/src/branch/master/roles/… seems to be the issue. This causes Airship's deckhand functional docker jobs to fail in pre-run, https://zuul.opendev.org/t/openstack/build/7493038ee1744465b2387b44e067d029…, and the jobs are retried. It is possible this is fallout from Zuul's default ansible version being bumped to 2.8 from 2.7. This also seems to affect OSH's kubernetes keystone auth job, https://zuul.opendev.org/t/openstack/build/68d937b3e5e3449cb6fe2e6947bbf0db… Ironic The Ironic example is a bit different and runs into some fun Ansible behaviors. Ironic's IPA multinode jobs (possibly others too) are filling the root disk of the test nodes on some clouds, http://paste.openstack.org/show/787641/. Ansible uses /tmp to bootstrap its ssh connectivity and when /tmp is on / and / is full that results in Ansible returning a network failure error code. This then causes Zuul to rerun the job. Unfortunately because Ansible thinks networking is broken we have to do special things in a Zuul cleanup playbook to double check disk usage and that info is only available on the Zuul executors. But I've double checked for you and can confirm this is what is happening. The fix here is to use the ephemeral disk which is mounted on /opt and contains much more disk space. Another option would be to reduce the size of Ironic's fake baremetal images. In any case they are aware of this and we should expect a fix soon. There are likely more cases I haven't found yet, but I wanted to point out this Zuul behavior to people. The intent is that it handle exceptional cases and not paper over failures that happen often or even 100% of the time. We should do our best to fix these problems when they pop up. Thank you to everyone that has already stepped up to help fix some of these! Clark

2 2

[glance] no meeting on 26 December 2019
by Abhishek Kekane 19 Dec '19

19 Dec '19

As discussed during yesterday's meeting, we will not hold the glance weekly meeting on 26th December 2019. The next meeting will be on 2 January 2020. Wish you all a happy christmas and new year in advance. Thanks & Best Regards, Abhishek Kekane

1 0

[ironic] - automated cleaning
by fsbiz＠yahoo.com 19 Dec '19

19 Dec '19

We have enabled automatic cleaning and I have noticed that our UEFI enabled nodes most often end up in "clean failed" state. Basic troubleshooting reveals that PXE correctly boots IPXE, but the IPXE goes and boots the OS instead of downloading the deployed_kernel / ramdisk. Here's the timeline of events.Dec 18 12:19:39 sc-ironic04 Node 56e58642-12ac-4455-bc95-2a328198f845 moved to provision state "cleaning"Dec 18 12:20:32 sc-ironic04 Successfully set node 56e58642-12ac-4455-bc95-2a328198f845 power state to power onDec 18 12:21:14 DHCPACK(ns-601d0738-39) 10.33.23.7 6c:b3:11:4f:8b:18 (PXE Boot gets IP: 10.33.23.7)Dec 18 12:21:15 sc-ironic04 in.tftpd[367896]: Client 10.33.23.7 finished ipxe_x86_64.efi (TFTP of IPXE is complete)Dec 18 12:21:23 DHCPACK(ns-261f35c5-7e) 10.33.23.7 6c:b3:11:4f:8b:18 host-10-33-23-7 (IPXE acquires IP: 10.33.23.7)NO HTTP request for deploy_kernel Instead:Dec 18 12:22:40 sc-control04 dnsmasq-dhcp[3508449]: 3990148443 DHCPREQUEST(ns-261f35c5-7e) 10.33.22.188 6c:b3:11:4f:8b:18Dec 18 12:22:40 sc-control04 dnsmasq-dhcp[3508449]: 3990148443 DHCPNAK(ns-261f35c5-7e) 10.33.22.188 6c:b3:11:4f:8b:18 wrong address When the OS was running it had IP address of 10.33.22.188. The OS came up and tries to renew its lease for 10.33.22.188 which was NAKed by the DHCP server. The OS then did a full DHCP (DISCOVER, etc.), got a valid IP address, did a cloud-init, etc. Is this a known issue that has been fixed ? Any pointers would be appreciated. thanks,Farad.

1 0

[magnum] Next weekly team meeting will be on 22 Jan 2020
by Feilong Wang 19 Dec '19

19 Dec '19

Hi team, We just had a good year and we deserve some rest. We will skip some meetings and the next one will be on 22 Jan 2020. Hope you all will have a merry Christmas and happy New Year. -- Cheers & Best regards, Feilong Wang (王飞龙) Head of R&D Catalyst Cloud - Cloud Native New Zealand -------------------------------------------------------------------------- Tel: +64-48032246 Email: flwang(a)catalyst.net.nz Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------

1 0

[qa][nova][cinder] ceph job has been busted on stable/pike for awhile, fix or delete?
by Matt Riedemann 19 Dec '19

19 Dec '19

The ceph job on stable/pike has been broken for awhile [1]. The fix to make devstack-plugin-ceph (which is branchless) work on stable/pike depends on this devstack change on stable/pike to introduce the TARGET_BRANCH variable [2]. Question is, can we move [2] forward? If not, I make a motion that we drop ceph jobs from stable/pike (and likely stable/ocata if we are still running them there) because they won't work. The ceph job is non-voting anyway so that's why it goes unnoticed for so long when it's broken. If we aren't going to fix this stuff we shouldn't be wasting CI resources running broken jobs. Did I mention that you should all get off my lawn? :) [1] https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1835627 [2] https://review.opendev.org/#/c/684756/ -- Thanks, Matt

5 7