- openstack-discuss - lists.openstack.org

[tc] Follow-up on action tems
by Jean-Philippe Evrard 21 Jan '20

21 Jan '20

Hello TC members, Here are a few things that should keep your attention, on top of the usual reviews... - We should start the business opportunities for 2020. Any takers? - We have action items from the last meeting: - ttx to post a summary of large scale sig's meeting on the ML this week. - mnaser to follow up on the static hosting - mnaser to push an update to distill what happened on the ML in terms of stable branch policy - jungleboyj to update the survey report patch - everyone to let the multi-arch SIG ( https://etherpad.openstack.org/p/Multi-arch) know if you have CI resources in different architectures - We have also new action items: - We should update the governance dashboard to include the "ideas" repo. Anyone? - We need to keep tracking on Telemetry status ( http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011586… ). - Our next meeting will host on 6 February. You can provide suggestion on agenda https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee - ttx should report on his investigation on possible bylaws change for an eventual leadership (tc/uc) merge, UC was more or less okay with the idea, which allows to continue forward the investigation. Note: There is no official stance from the TC on this yet. It's just investigation. - We have updated SIGs tags with https://review.opendev.org/#/c/695625/. This will make sure people aware of the actual status. And as mentioned action ttx and ricolin will works on expose that status tag on governance-sigs' page. And finally, some things to keep in mind: - We need to promote the first contact sig to organisations that might need help on getting things achieved in OpenStack - We need to start collecting ideas for the ideas repo. Regards, JP & Rico

1 0

[all][tc] What happened in OpenStack Governance recently
by Jean-Philippe Evrard 21 Jan '20

21 Jan '20

Hello everyone, It's official, the V release will be codenamed Victoria! The TC is now working on declaring its runtimes [A]. The TC has merged a change in its charter, establishing new rules for the future terms [B]. This will give us leeway for elections and term lengths. The cycle goals selection schedule was made clearer, so that we don't add goals to the teams too late in the future [C]. We have selected the 'project specific PTL and contributor guides' as an official goal for Ussuri. I invite you to read the update of Ghanshyam Mann (gmann) to follow what's going on for our Ussuri goals [D]. You should also have a look at the zuul legacy migration [G], as it could eventually be selected as a V goal :) In terms of release naming, we'll start the W naming process soon, because we know that naming things are hard [E]. If you're interested by security policy, you can have a look at Jeremy's patch [F]. In terms of projects: - There is a new official charm, for watcher [H] - js-openstack-lib is now part of the openstacksdk team [I] - there is a new neutron project, ovn-octavia-provider [J] - the ansible-sig got a new repository, ansible-plugin-container- connection [K] - Justin Ferrieu is the PTL for Cloudkitty for the rest of the cycle [L] - sushi-cli is now part of the ironic team [M] Happy new year to our Chinese community! Regards, Jean-Philippe & Rico [A]: https://review.opendev.org/#/c/693743/ [B]: https://review.opendev.org/#/c/699277/ [C]: https://review.opendev.org/#/c/698299/ [D]: http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012019.… [E]: https://review.opendev.org/#/c/702414/ [F]: https://review.opendev.org/#/c/678426/ [G]: https://review.opendev.org/#/c/691278/ [H]: https://review.opendev.org/#/c/702072/ [I]: https://review.opendev.org/#/c/701854/ [J]: https://review.opendev.org/#/c/697095/ [K]: https://review.opendev.org/#/c/696737/ [L]: https://review.opendev.org/#/c/701214/ [M]: https://review.opendev.org/#/c/700914/

1 0

[all][ci] Gate jobs failures because of broken pip
by Takashi Kajinami 21 Jan '20

21 Jan '20

Hi All, I now observe some of the gate jobs are failing[1] because of broken pip[2], and it guess that we need to pin pip to avoid the error atm. [1] https://81b633e2c5fe858f8400-d324a81a71d524d51ede3dc5aee27774.ssl.cf5.rackc… [2] https://github.com/pypa/pip/issues/7217 May I ask for some help to set that pinning ? I couldn't find the correct way to pin pip after several investigation ... Thank you, Takashi -- ---------- Takashi Kajinami Software Maintenance Engineer Customer Experience and Engagement Red Hat e-mail: tkajinam(a)redhat.com

3 2

[tripleo][ironic][ansible][openstack-ansible] Ironic/Baremetal Ansible modules
by Sagi Shnaidman 21 Jan '20

21 Jan '20

Hi, all in the light of finding the new home place for openstack related ansible modules [1] I'd like to discuss the best strategy to create Ironic ansible modules. Existing Ironic modules in Ansible repo don't cover even half of Ironic functionality, don't fit current needs and definitely require an additional work. There are a few topics that require attention and better be solved before modules are written to save additional work. We prepared an etherpad [2] with all these questions and if you have ideas or suggestions on how it should look you're welcome to update it. We'd like to decide the final place for them, name conventions (the most complex one!), what they should look like and how better to implement. Anybody interested in Ansible and baremetal management in Openstack, you're more than welcome to contribute. Thanks [1] https://review.opendev.org/#/c/684740/ [2] https://etherpad.openstack.org/p/ironic-ansible-modules -- Best regards Sagi Shnaidman

9 18

[horizon][cloudkitty][tacker][masakari][congress][karbar][trove] Drop python2.7 and Django1.11 support in horizon plugins
by Akihiro Motoki 21 Jan '20

21 Jan '20

Hi, I am sending this mail to encourage reviews in horizon plugins. - Drop python 2.7 and Django 1.11 - Remove six usage Horizon has dropped python 2.7 support and Django 1.11 support (which is the last version with python 2.7 support). We proposed changes to horizon plugins to catch up with these changes. - Most projects have merged them. Thanks for your cooperation. - There are several pending reviews. https://review.opendev.org/#/q/topic:drop-django111-support+status:open - Django 1.11 drop depends on python 2.7 drop. In case of karbar-dashboard, python 2.7 drop needs to land before it. https://review.opendev.org/#/c/694442/ horizon also has dropped six usage. There are plugins which depends six on horizon requirements. We proposed a series of changes to avoid UT breakages in plugins. Reviews on them are also appreciated. https://review.opendev.org/#/q/topic:remove-six+status:open Thanks, Akihiro Motoki (amotoki)

1 0

[queens][nova] iscsi issue
by Ignazio Cassano 20 Jan '20

20 Jan '20

Hello all we are testing openstack queens cinder driver for Unity iscsi (driver cinder.volume.drivers.dell_emc.unity.Driver). The unity storage is a Unity600 Version 4.5.10.5.001 We are facing an issue when we try to detach volume from a virtual machine with two or more volumes attached (this happens often but not always): The following is reported nova-compute.log: 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server self.connector.disconnect_volume(connection_info['data'], None) 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 848, in disconnect_volume 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server device_info=device_info) 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 892, in _cleanup_connection 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server path_used, was_multipath) 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py", line 271, in remove_connection 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server self.flush_multipath_device(multipath_name) 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py", line 329, in flush_multipath_device 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server root_helper=self._root_helper) 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs) 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server return execute_root(*cmd, **kwargs) 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs) 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server raise exc_type(*result[2]) 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command. 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server Command: multipath -f 36006016006e04400d0c4215e3ec55757 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server Exit code: 1 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server Stdout: u'Jan 17 16:04:30 | 36006016006e04400d0c4215e3ec55757p1: map in use\nJan 17 16:04:31 | failed to remove multipath map 36006016006e04400d0c4215e3ec55757\n' 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server Stderr: u'' 2020-01-17 16:05:11.132 6643 ERROR oslo_messaging.rpc.server Best Regards Ignazio

2 2

[magnum][kolla] etcd wal sync duration issue
by Eric K. Miller 20 Jan '20

20 Jan '20

Hi, We are using the following coe cluster template and cluster create commands on an OpenStack Stein installation that installs Magnum 8.2.0 Kolla containers installed by Kolla-Ansible 8.0.1: openstack coe cluster template create \ --image Fedora-AtomicHost-29-20191126.0.x86_64_raw \ --keypair userkey \ --external-network ext-net \ --dns-nameserver 1.1.1.1 \ --master-flavor c5sd.4xlarge \ --flavor m5sd.4xlarge \ --coe kubernetes \ --network-driver flannel \ --volume-driver cinder \ --docker-storage-driver overlay2 \ --docker-volume-size 100 \ --registry-enabled \ --master-lb-enabled \ --floating-ip-disabled \ --fixed-network KubernetesProjectNetwork001 \ --fixed-subnet KubernetesProjectSubnet001 \ --labels kube_tag=v1.15.7,cloud_provider_tag=v1.15.0,heat_container_agent_tag=ste in-dev,master_lb_floating_ip_enabled=true \ k8s-cluster-template-1.15.7-production-private openstack coe cluster create \ --cluster-template k8s-cluster-template-1.15.7-production-private \ --keypair userkey \ --master-count 3 \ --node-count 3 \ k8s-cluster001 The deploy process works perfectly, however, the cluster health status flips between healthy and unhealthy. The unhealthy status indicates that etcd has an issue. When logged into master-0 (out of 3, as configured above), "systemctl status etcd" shows the stdout from etcd, which shows: Jan 11 17:27:36 k8s-cluster001-4effrc2irvjq-master-0.novalocal runc[2725]: 2020-01-11 17:27:36.548453 W | etcdserver: timed out waiting for read index response Jan 11 17:28:02 k8s-cluster001-4effrc2irvjq-master-0.novalocal runc[2725]: 2020-01-11 17:28:02.960977 W | wal: sync duration of 1.696804699s, expected less than 1s Jan 11 17:28:31 k8s-cluster001-4effrc2irvjq-master-0.novalocal runc[2725]: 2020-01-11 17:28:31.292753 W | wal: sync duration of 2.249722223s, expected less than 1s We also see: Jan 11 17:40:39 k8s-cluster001-4effrc2irvjq-master-0.novalocal runc[2725]: 2020-01-11 17:40:39.132459 I | etcdserver/api/v3rpc: grpc: Server.processUnaryRPC failed to write status: stream error: code = DeadlineExceeded desc = "context deadline exceeded" We initially used relatively small flavors, but increased these to something very large to be sure resources were not constrained in any way. "top" reported no CPU nor memory contention on any nodes in either case. Multiple clusters have been deployed, and they all have this issue, including empty clusters that were just deployed. I see a very large number of reports of similar issues with etcd, but discussions lead to disk performance, which can't be the cause here, not only because persistent storage for etcd isn't configured in Magnum, but also the disks are "very" fast in this environment. Looking at "vmstat -D" from within master-0, the number of writes is minimal. Ceilometer logs about 15 to 20 write IOPS for this VM in Gnocchi. Any ideas? We are finalizing procedures to upgrade to Train, so we wanted to be sure that we weren't running into some common issue with Stein that would immediately be solved with Train. If so, we will simply proceed with the upgrade and avoid diagnosing this issue further. Thanks! Eric

4 6

Galera config values
by Albert Braden 20 Jan '20

20 Jan '20

I'm experimenting with Galera in my Rocky openstack-ansible dev cluster, and I'm finding that the default haproxy config values don't seem to work. Finding the correct values is a lot of work. For example, I spent this morning experimenting with different values for "timeout client" in /etc/haproxy/haproxy.cfg. The default is 1m, and with the default set I see this error in /var/log/nova/nova-scheduler.log on the controllers: 2020-01-17 13:54:26.059 443358 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) There are several timeout values in /etc/haproxy/haproxy.cfg. These are the values we started with: stats timeout 30s timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout check 10s At first I changed them all to 30m. This stopped the "Lost connection" error in nova-scheduler.log. Then, one at a time, I changed them back to the default. When I got to "timeout client" I found that setting it back to 1m caused the errors to start again. I changed it back and forth and found that 4 minutes causes errors, and 6m stops them, so I left it at 6m. These are my active variables: root@us01odc-dev2-ctrl1:/etc/mysql# mysql -e 'show variables;'|grep timeout connect_timeout 20 deadlock_timeout_long 50000000 deadlock_timeout_short 10000 delayed_insert_timeout 300 idle_readonly_transaction_timeout 0 idle_transaction_timeout 0 idle_write_transaction_timeout 0 innodb_flush_log_at_timeout 1 innodb_lock_wait_timeout 50 innodb_rollback_on_timeout OFF interactive_timeout 28800 lock_wait_timeout 86400 net_read_timeout 30 net_write_timeout 60 rpl_semi_sync_master_timeout 10000 rpl_semi_sync_slave_kill_conn_timeout 5 slave_net_timeout 60 thread_pool_idle_timeout 60 wait_timeout 3600 So it looks like the value of "timeout client" in haproxy.cfg needs to match or exceed the value of "wait_timeout" in mysql. Also in nova.conf I see "#connection_recycle_time = 3600" - I need to experiment to see how that value interacts with the timeouts in the other config files. Is this the best way to find the correct config values? It seems like there should be a document that talks about these timeouts and how to set them (or maybe more generally how the different timeout settings in the various config files interact). Does that document exist? If not, maybe I could write one, since I have to figure out the correct values anyway.

4 5

[openstack-helm] Core Reviewer Nominations
by Pete Birley 20 Jan '20

20 Jan '20

OpenStack-Helm team, Based on their record of quality code review and substantial/meaningful code contributions to the openstack-helm project, at last weeks meeting we proposed the following individuals as core reviewers for openstack-helm: - Gage Hugo - Steven Fitzpatrick All OpenStack-Helm Core Reviewers are invited to reply with a +1/-1 by EOD next Monday (27/1/2020). A lone +1/-1 will apply to both candidates, otherwise please spell out votes individually for the candidates. Cheers, Pete

2 1

[neutron] Bug deputy report for week of January 13th
by Brian Haley 20 Jan '20

20 Jan '20

Hi, I was Neutron bug deputy last week. Below is a short summary about reported bugs. -Brian Critical bugs ------------- * https://bugs.launchpad.net/neutron/+bug/1859988 - neutron-tempest-plugin tests fail for stable/queens - bcafarel picked up, related to below bug * https://bugs.launchpad.net/neutron/+bug/1860033 - neutron tempest jobs broken on rocky due to requirements neutron-lib upgrade - many proposed changes, discussion on ML High bugs --------- * https://bugs.launchpad.net/neutron/+bug/1859977 - [OVN] Update of floatingip creates new row in NBDB NAT table - maciej picked up * https://bugs.launchpad.net/neutron/+bug/1860140 - [OVN] Provider driver sends malformed update to Octavia - https://review.opendev.org/#/c/703097 * https://bugs.launchpad.net/neutron/+bug/1860141 - [OVN] Provider driver fails while LB VIP port already created - https://review.opendev.org/#/c/703110 Medium bugs ----------- * https://bugs.launchpad.net/neutron/+bug/1859832 - L3 HA connectivity to GW port can be broken after reboot of backup node - Issue with MLDv2 packet causing issues with connections - Slawek picked up ownership * https://bugs.launchpad.net/neutron/+bug/1860273 - https://review.opendev.org/#/c/703292/ Low bugs -------- * https://bugs.launchpad.net/neutron/+bug/1859765 - Sgs of device_owner_network port can be update without specifing "device_owner" attr - https://review.opendev.org/#/c/702632/ - discussion in review * https://bugs.launchpad.net/neutron/+bug/1859962 - Sanity checks comparing versions should not use decimal comparison - https://review.opendev.org/#/c/702847/ Wishlist bugs ------------- Invalid bugs ------------ * https://bugs.launchpad.net/neutron/+bug/1859638 - VIP between dvr east-west networks does not work at all - Duplicate of https://bugs.launchpad.net/neutron/+bug/1774459 * https://bugs.launchpad.net/neutron/+bug/1859976 - Removing smart_nic port in openstack will try to delete representor port - os-vif issue Further triage required ----------------------- * https://bugs.launchpad.net/neutron/+bug/1859362 - Neutron accepts arbitrary MTU values for networks - Rodolfo has some questions on the bug * https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1859649 - networking disruption on upgrade from 14.0.0 to 14.0.3 - Rodolfo had question about restart order of services * https://bugs.launchpad.net/neutron/+bug/1859887 - External connectivity broken because of stale FIP rule - Asked for release information, etc * https://bugs.launchpad.net/networking-ovn/+bug/1859965 - networking-ovn's octavia api，how get octavia-api DB use python code? - networking-ovn octavia driver bug? asked for more information

2 1

[release] Python universal wheel support
by Sean McGinnis 20 Jan '20

20 Jan '20

Greetings, We have just merged a change to the release-openstack-python job that changes the wheels we produce to not be universal. For some background - we added an explicit flag of "--universal" to the creation of wheels a while back. For projects that have both Python 2 and 3 support, you want an universal wheel. Not all (probably most) projects did not add this flag to their setup.cfg, so overriding at the release job level was considered a good way to make sure our output was what we wanted at the time. We now have the majority of projects dropping py2 support, so we actually no longer want to create these universal wheels if py2 support has been dropped. That has actually been seen to cause some issues. The downside with this change is that the job is for *all* deliverables we release, including stable releases. So with this change, any new stable branches will no longer get universal wheels if the flag has not been set locally. This was deemed a good tradeoff with the current needs though. The lack of a univeral wheel may just make installation of py2 stable deliverables just slightly slower, but should not cause any real issues. Actions ------ Most likely this won't require any actions on the project team's part. If you have a project that still supports both py2 and py3 and do not have the flag set in setup.cfg, that can be added to still get the universal wheels built. That is done by adding the following: ``` [bdist_wheel] universal = 1 ``` Again, the performance impact is probably very minimal during installation, so this shouldn't be a major concern. If there are any oddities noticed after this change, please bring them up and we can help investigate what is happening. Thanks! Sean

1 0

[Multi-Arch-sig] Meeting this week
by Rico Lin 20 Jan '20

20 Jan '20

Hi all. As reminder, we will host our meeting this week. Tuesday at 0800 UTC in #openstack-meeting-alt and 1500 UTC in #openstack-meeting . Feel free to propose agenda here https://etherpad.openstack.org/p/Multi-Arch-agenda See you all in meeting:) -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

1 0

Cinder snapshot delete successful when expected to fail
by Tony Pearce 20 Jan '20

20 Jan '20

Could anyone help by pointing me where to go to be able to dig into this issue further? I have installed a test Openstack environment using RDO Packstack. I wanted to install the same version that I have in Production (Pike) but it's not listed in the CentOS repo via yum search. So I installed Queens. I am using nimble.py Cinder driver. Nimble Storage is a storage array accessed via iscsi from the Openstack host, and is controlled from Openstack by the driver and API. *What I expected to happen:* 1. create an instance with volume (the volume is created on the storage array successfully and instance boots from it) 2. take a snapshot (snapshot taken on the volume on the array successfully) 3. create a new instance from the snapshot (the api tells the array to clone the snapshot into a new volume on the array and use that volume for the instance) 4. try and delete the snapshot Expected Result - Openstack gives the user a message like "you're not allowed to do that". Note: Step 3 above creates a child volume from the parent snapshot. It's impossible to delete the parent snapshot because IO READ is sent to that part of the original volume (as I understand it). *My production problem is this: * 1. create an instance with volume (the volume is created on the storage array successfully) 2. take a snapshot (snapshot taken on the volume on the array successfully) 3. create a new instance from the snapshot (the api tells the array to clone the snapshot into a new volume on the array and use that volume for the instance) 4. try and delete the snapshot Result - snapshot goes into error state and later, all Cinder operations fail such as new instance/create volume etc. until the correct service is restarted. Then everything works once again. To troubleshoot the above, I installed the RDP Packstack Queens (because I couldnt get Pike). I tested the above and now, the result is the snapshot is successfully deleted from openstack but not deleted on the array. The log is below for reference. But I can see the in the log that the array sends back info to openstack saying the snapshot has a clone and the delete cannot be done because of that. Also response code 409. *Some info about why the problem with Pike started in the first place* 1. Vendor is Nimble Storage which HPE purchased 2. HPE/Nimble have dropped support for openstack. Latest supported version is Queens and Nimble array version v4.x. The current Array version is v5.x. Nimble say there are no guarantees with openstack, the driver and the array version v5.x 3. I was previously advised by Nimble that the array version v5.x will work fine and so left our DR array on v5.x with a pending upgrade that had a blocker due to an issue. This issue was resolved in December and the pending upgrade completed to match the DR array took place around 30 days ago. With regards to the production issue, I assumed that the array API has some changes between v4.x and v5.x and it's causing an issue with Cinder due to the API response. Although I have not been able to find out if or what changes there are that may have occurred after the array upgrade, as the documentation for this is Nimble internal-only. *So with that - some questions if I may:* When Openstack got the 409 error response from the API (as seen in the log below), why would Openstack then proceed to delete the snapshot on the Openstack side? How could I debug this further? I'm not sure what Openstack Cinder is acting on in terns of the response as yet. Maybe Openstack is not specifically looking for the error code in the response? The snapshot that got deleted on the openstack side is a problem. Would this be related to the driver? Could it be possible that the driver did not pass the error response to Cinder? Thanks in advance. Just for reference, the log snippet is below. ==> volume.log <== > 2020-01-17 16:53:23.718 24723 WARNING py.warnings > [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 > 87e34c89e6fb41d2af25085b64011a55 - default default] > /usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:852: > InsecureRequestWarning: Unverified HTTPS request is being made. Adding > certificate verification is strongly advised. See: > https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings > InsecureRequestWarning) > : NimbleAPIException: Failed to execute api > snapshots/0464a5fd65d75fcfe1000000000000011100001a41: Error Code: 409 > Message: Snapshot snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 for volume > volume-5b02db35-8d5c-4ef6-b0e7-2f9b62cac57e has a clone. > ==> api.log <== > 2020-01-17 16:53:23.769 25242 INFO cinder.api.openstack.wsgi > [req-bfcbff34-134b-497e-82b1-082d48f8767f df7548ecad684f26b8bc802ba63a9814 > 87e34c89e6fb41d2af25085b64011a55 - default default] > http://192.168.53.45:8776/v3/87e34c89e6fb41d2af25085b64011a55/volumes/detail > returned with HTTP 200 > 2020-01-17 16:53:23.770 25242 INFO eventlet.wsgi.server > [req-bfcbff34-134b-497e-82b1-082d48f8767f df7548ecad684f26b8bc802ba63a9814 > 87e34c89e6fb41d2af25085b64011a55 - default default] 192.168.53.45 "GET > /v3/87e34c89e6fb41d2af25085b64011a55/volumes/detail HTTP/1.1" status: 200 > len: 4657 time: 0.1152730 > ==> volume.log <== > 2020-01-17 16:53:23.811 24723 WARNING py.warnings > [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 > 87e34c89e6fb41d2af25085b64011a55 - default default] > /usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:852: > InsecureRequestWarning: Unverified HTTPS request is being made. Adding > certificate verification is strongly advised. See: > https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings > InsecureRequestWarning) > : NimbleAPIException: Failed to execute api > snapshots/0464a5fd65d75fcfe1000000000000011100001a41: Error Code: 409 > Message: Snapshot snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 for volume > volume-5b02db35-8d5c-4ef6-b0e7-2f9b62cac57e has a clone. > 2020-01-17 16:53:23.902 24723 ERROR cinder.volume.drivers.nimble > [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 > 87e34c89e6fb41d2af25085b64011a55 - default default] Re-throwing Exception > Failed to execute api snapshots/0464a5fd65d75fcfe1000000000000011100001a41: > Error Code: 409 Message: Snapshot > snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 for volume > volume-5b02db35-8d5c-4ef6-b0e7-2f9b62cac57e has a clone.: > NimbleAPIException: Failed to execute api > snapshots/0464a5fd65d75fcfe1000000000000011100001a41: Error Code: 409 > Message: Snapshot snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 for volume > volume-5b02db35-8d5c-4ef6-b0e7-2f9b62cac57e has a clone. > 2020-01-17 16:53:23.903 24723 WARNING cinder.volume.drivers.nimble > [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 > 87e34c89e6fb41d2af25085b64011a55 - default default] Snapshot > snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 : has a clone: > NimbleAPIException: Failed to execute api > snapshots/0464a5fd65d75fcfe1000000000000011100001a41: Error Code: 409 > Message: Snapshot snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 for volume > volume-5b02db35-8d5c-4ef6-b0e7-2f9b62cac57e has a clone. > 2020-01-17 16:53:23.964 24723 WARNING cinder.quota > [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 > 87e34c89e6fb41d2af25085b64011a55 - default default] Deprecated: Default > quota for resource: snapshots_Nimble-DR is set by the default quota flag: > quota_snapshots_Nimble-DR, it is now deprecated. Please use the default > quota class for default quota. > 2020-01-17 16:53:24.054 24723 INFO cinder.volume.manager > [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 > 87e34c89e6fb41d2af25085b64011a55 - default default] Delete snapshot > completed successfully. Regards, *Tony Pearce* | *Senior Network Engineer / Infrastructure Lead**Cinglevue International <https://www.cinglevue.com>* Email: tony.pearce(a)cinglevue.com Web: http://www.cinglevue.com *Australia* 1 Walsh Loop, Joondalup, WA 6027 Australia. Direct: +61 8 6202 0036 | Main: +61 8 6202 0024 Note: This email and all attachments are the sole property of Cinglevue International Pty Ltd. (or any of its subsidiary entities), and the information contained herein must be considered confidential, unless specified otherwise. If you are not the intended recipient, you must not use or forward the information contained in these documents. If you have received this message in error, please delete the email and notify the sender.

3 4

[nova] Nova CI busted, please hold rechecks
by Eric Fried 19 Jan '20

19 Jan '20

The nova-live-migration job is failing 100% since yesterday morning [1]. Your rechecks won't work until that's resolved. I'll send an all-clear message when we're green again. Thanks, efried [1] https://bugs.launchpad.net/nova/+bug/1860021

3 3

DR options with openstack
by Tony Pearce 18 Jan '20

18 Jan '20

7 13

[tc][infra] Splitting OpenDev out of OpenStack Governance
by Clark Boylan 17 Jan '20

17 Jan '20

Hello, About six weeks ago I kicked off a discussion on what the future of OpenDev's governance looks like [0]. I think we had expected part of this process would be to split OpenDev out of OpenStack's governance, but wanted to be sure we had a bit more of a plan before we made that official. After some discussion in this thread [0] I believe we now have enough of a plan to move forward on splitting out. I've pushed a change [1] to the openstack/governance repo to make this official in git. I also wanted to make sure this change had some visibility so am sending this email too. Now for some background. The OpenDev effort intends to make our software development tools and processes available to projects outside of the OpenStack itself. We have actually made these resources available since Stackforge, but one of the major concerns we hear over and over is the implication that a project hosted on our platforms is still "OpenStack". The next step to avoiding this confusion and better reflecting our goals is to formally remove OpenDev from OpenStack's governance. As mentioned in the original thread [0], OpenDev would still incorporate input from the OpenStack project as one of its users. We aren't going away and will continue to work closely together to meet OpenStack's needs. But now we'll formalize doing that with other projects as well. Feedback is still welcome, though I ask people to read through the original thread [0] first. Please let me know if there is anything else I can do to help with this process. [0] http://lists.openstack.org/pipermail/openstack-infra/2019-December/006537.h… [1] https://review.opendev.org/703134 Clark

1 0

[Neutron] cancel neutron L3 meeting
by LIU Yulong 17 Jan '20

17 Jan '20

Hi all, Hi guys, due to the Chinese Spring Festival I will be offline in next two weeks. So I will not be available to chair the L3 meeting. Let's cancel the next two meetings. Then the L3 meeting will be rescheduled on 5th Feb, 2020. OK, see you guys then.  And happy Chinese New Year! 春节快乐！ Regards, LIU Yulong

1 0

Re: [Release-job-failures] release-post job for openstack/releases for ref refs/heads/master failed
by Thierry Carrez 17 Jan '20

17 Jan '20

zuul(a)openstack.org wrote: > Build failed. > > - tag-releases https://zuul.opendev.org/t/openstack/build/a414023508294a65abe9715546757e41 : POST_FAILURE in 5m 13s > - publish-tox-docs-static https://zuul.opendev.org/t/openstack/build/None : SKIPPED There was an error running the post-job tasks on the tag job on https://review.opendev.org/702925. While trying to collect log output: ssh: connect to host 38.108.68.119 port 22: No route to host This looks like a transient error, and it can be ignored (the job itself had run and was a NOOP anyway). -- Thierry Carrez (ttx)

1 0

[cinder][ops] new driver/new target driver merge deadline
by Brian Rosmaita 17 Jan '20

17 Jan '20

Greetings to anyone developing a new Cinder driver (or target driver), or anyone trying to get someone to develop such a driver, This is a reminder that the deadline for merging a new backend driver or a new target driver to Cinder for the Ussuri release is the Ussuri-2 milestone on 13 February 2020 (23:59 UTC). New drivers must be (a) code complete including unit tests, (b) merged into the code repository, and (c) must have a 3rd Party CI running reliably. (The idea is that new drivers will be included in a release at the second milestone and thus be easily available for downstream testing, documentation feedback, etc.) You can find more information about Cinder drivers here: https://docs.openstack.org/cinder/latest/drivers-all-about.html and you can ask questions in #openstack-cinder on IRC or here on the mailing list. cheers, brian

1 0

[magnum] subnet created in public network?
by Ionut Biru 17 Jan '20

17 Jan '20

Hello, I'm using magnum 9.2.0 and while trying to experiment with this version, i was finding out that while deploying the cluster, heat creates the subnet into the public network. In the past, on rocky and stein, magnum/heat was creating a new network, with a router and an port within the public network for connectivity. I was wondering, if this is the expected behavior (subnet in public network). How do I revert to the old way of having new network? -- Ionut Biru - https://fleio.com

1 0

[largescale-sig] Meeting summary and next actions
by Thierry Carrez 17 Jan '20

17 Jan '20

Hi everyone, The Large Scale SIG held a meeting earlier this week. Thanks to belmiro for chairing it! You can access the summary and logs of the meeting at: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_si… For the "Scaling within one cluster, and instrumentation of the bottlenecks" goal, I created a ML thread and etherpad to collect user stories, so far without much success. masahito is still working on the draft for oslo.metrics, hopefully will be ready by end of January. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011925.… [2] https://etherpad.openstack.org/p/scaling-stories Standing TODOs: - all post short descriptions of what happens (what breaks first) when scaling up a single cluster to https://etherpad.openstack.org/p/scaling-stories - masahito to produce first draft for the oslo.metric blueprint - all learn more about golden signals concept as described in https://landing.google.com/sre/book.html For the "Document large scale configuration and tips &tricks" goal, amorin started a thread[3] and etherpad[4] on documenting configuration defaults for large scale, to which slaweq contributed for Neutron. [3] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011820.… [4] https://etherpad.openstack.org/p/large-scale-sig-documentation Standing TODOs: - oneswig to follow up with Scientific community to find articles around large scale openstack The next meeting will happen on January 29, at 9:00 UTC on #openstack-meeting. They will happen from now on every two weeks. Cheers, -- Thierry Carrez (ttx)

1 0

RE: DR options with openstack
by Burak Hoban 17 Jan '20

17 Jan '20

Hey Tony, Keep in mind that if you're looking to run OpenStack, but you're not feeling comfortable with the community support then there's always the option to go with a vendor backed version. These are usually a good option for those a little more risk adverse, or who don't have the time/or skills to maintain upstream releases - however going down that path usually means you can do less with OpenStack (depending on the vendor), but you have a large pool of resources to help troubleshoot and answer questions. We do both approaches internally for different clusters, so both approaches have their pro and cons. You touched on a few points in your original email... > If you had two OpenStack clusters, one in "site 1" and another in "site 2", then you could look at below for backup/restore of instances cross-cluster: - Freezer -> https://wiki.openstack.org/wiki/Freezer - Trillio (basically just a series of nova snapshots under the cover) -> https://www.trilio.io/ You could then over the top roll out a file level based backup tool on each instance, this would pretty much offer you replication functionality without having to do block-level tinkering. > Failover of OpenStack controller/computes If you have two sites, you can always go for 3x Controller deployment spanning cross site. Depending on latency obviously, however all you really need is a good enough link for RabbitMQ/Galera to talk reliably etc. Failing that, I'd recommend backing up your Controller with ReaR. From there you can also schedule frequent automated jobs to do a OpenStack DB backups. Recovering should be a case of ReaR restore, load latest OpenStack DB and start everything up... You'll probably want to ensure your VLANs are spanned cross-site so you can reuse same IP addresses. https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/post_de… https://superuser.openstack.org/articles/tutorial-rear-openstack-deployment/ In reality, the best solution would be to have two isolated clusters, and your workloads spanned across both sites. Obviously that isn't always possible (from personal experience), but pushing people down the Kubernetes path and then for the rest automation/backup utilities may cater for your needs. Having said that, Albert's link does look promising -> https://docs.openstack.org/cinder/pike/contributor/replication.html Date: Thu, 16 Jan 2020 19:49:08 +0000 From: Albert Braden <Albert.Braden(a)synopsys.com> To: Tony Pearce <tony.pearce(a)cinglevue.com>, "openstack-discuss(a)lists.openstack.org" <openstack-discuss(a)lists.openstack.org> Subject: RE: DR options with openstack Message-ID: <BN8PR12MB3636451FC8E2BC6A50216425D9360(a)BN8PR12MB3636.namprd12.prod.outlook.com> Content-Type: text/plain; charset="utf-8" Hi Tony, It looks like Cheesecake didn’t survive but apparently some components of it did; details in https://docs.openstack.org/cinder/pike/contributor/replication.html I’m not using Cinder now; we used it at eBay with Ceph and Netapp backends. Netapp makes it easy but is expensive; Ceph is free but you have to figure out how to make it work. You’re right about forking; we did it and then upgrading turned from an incredibly difficult ordeal to an impossible one. It’s better to stay with the “official” code so that upgrading remains an option. I’m just an operator; hopefully someone more expert will reply with more useful info. It’s true that our community lacks participation. It’s very difficult for a new operator to start using openstack and get help with the issues that they encounter. So far this mailing list has been the best resource for me. IRC and Ask Openstack are mostly unattended. I try to help out in #openstack when I can, but I don’t know a lot so I mostly end up telling people to ask on the list. On IRC sometimes I find help by asking in other openstack-* channels. Sometimes people complain that I’m asking in a developer channel, but sometimes I get help. Persistence is the key. If I keep asking long enough in enough places, eventually someone will answer. If all else fails, I open a bug. Good luck and welcome to the Openstack community! From: Tony Pearce <tony.pearce(a)cinglevue.com> Sent: Wednesday, January 15, 2020 11:37 PM To: openstack-discuss(a)lists.openstack.org Subject: DR options with openstack Hi all My questions are; 1. How are people using iSCSI Cinder storage with Openstack to-date? For example a Nimble Storage array backend. I mean to say, are people using backend integration drivers for other hardware (like netapp)? Or are they using backend iscsi for example? 2. How are people managing DR with Openstack in terms of backend storage replication to another array in another location and continuing to use Openstack? The environment which I am currently using; 1 x Nimble Storage array (iSCSI) with nimble.py Cinder driver 1 x virtualised Controller node 2 x physical compute nodes This is Openstack Pike. In addition, I have a 2nd Nimble Storage array in another location. To explain the questions I’d like to put forward my thoughts for question 2 first: For point 2 above, I have been searching for a way to utilise replicated volumes on the 2nd array from Openstack with existing instances. For example, if site 1 goes down how would I bring up openstack in the 2nd location and boot up the instances where their volumes are stored on the 2nd array. I found a proposal for something called “cheesecake” ref: https://specs.openstack.org/openstack/cinder-specs/specs/rocky/cheesecake-p…<https://urldefense.proofpoint.com/v2/url?u=https-3A__specs.openstack.org_op…> But I could not find if it had been approved or implemented. So I return to square 1. I have some thoughts about failing over the controller VM and compute node but I don’t think there’s any need to go into here because of the above blocker and for brevity anyway. The nimble.py driver which I am using came with Openstack Pike and it appears Nimble / HPE are not maintaining it any longer. I saw a commit to remove nimble.py in Openstack Train release. The driver uses the REST API to perform actions on the array. Such as creating a volume, downloading the image, mounting the volume to the instance, snapshots, clones etc. This is great for me because to date I have around 10TB of openstack storage data allocated and the Nimble array shows the amount of data being consumed is <900GB. This is due to the compression and zero-byte snapshots and clones. So coming back to question 2 – is it possible? Can you drop me some keywords that I can search for such as an Openstack component like Cheesecake? I think basically what I am looking for is a supported way of telling Openstack that the instance volumes are now located at the new / second array. This means a new cinder backend. Example, new iqn, IP address, volume serial number. I think I could probably hack the cinder db but I really want to avoid that. So failing the above, it brings me to the question 1 I asked before. How are people using Cinder volumes? May be I am going about this the wrong way and need to take a few steps backwards to go forwards? I need storage to be able to deploy instances onto. Snapshots and clones are desired. At the moment these operations take less time than the horizon dashboard takes to load because of the waiting API responses. When searching for information about the above as an end-user / consumer I get a bit concerned. Is it right that Openstack usage is dropping? There’s no web forum to post questions. The chatroom on freenode is filled with ~300 ghosts. Ask Openstack questions go without response. Earlier this week (before I found this mail list) I had to use facebook to report that the Openstack.org website had been hacked. Basically it seems that if you’re a developer that can write code then you’re in but that’s it. I have never been a coder and so I am somewhat stuck. Thanks in advance Sent from Mail<https://urldefense.proofpoint.com/v2/url?u=https-3A__go.microsoft.com_fwlin…> for Windows 10 _____________________________________________________________________ The information transmitted in this message and its attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information, by persons or entities other than the intended recipient is prohibited. If you have received this in error, please contact the sender and delete this e-mail and associated material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files, with the permission of the sender. This message has been scanned for viruses. _____________________________________________________________________

6 6

[keystone] Keystone Team Update - Week of 13 January 2020
by Vishakha Agarwal 16 Jan '20

16 Jan '20

# Keystone Team Update - Week of 13 January 2020 ## News ### Roadmap Review The Team has decided to review the roadmap every other week so as to keep up the development momentum of the ussuri cycle [1]. [1] https://tree.taiga.io/project/keystone-ussuri-roadmap/kanban ### User Support and Bug Duty Every week the duty is being rotated between the members. The person-in-charge for bug duty for current and upcoming week can be seen on the etherpad [2] [2] https://etherpad.openstack.org/p/keystone-l1-duty ## Open Specs Ussuri specs: https://bit.ly/2XDdpkU Ongoing specs: https://bit.ly/2OyDLTh ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 7 changes this week. ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 36 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ### Priority Reviews * Community Goals https://review.opendev.org/#/c/699127/ [ussuri][goal] Drop python 2.7 support and testing keystone-tempest-plugin https://review.opendev.org/#/c/699119/ [ussuri][goal] Drop python 2.7 support and testing python-keystoneclient * Special Requests https://review.opendev.org/#/c/662734/ Change the default Identity endpoint to internal https://review.opendev.org/#/c/699013/ Always have username in CADF initiator https://review.opendev.org/#/c/700826/ Fix role_assignments role.id filter https://review.opendev.org/#/c/697444/ Adding options to user cli https://review.opendev.org/#/c/702374/ Cleanup doc/requirements.txt ## Bugs This week we opened 2 new bugs and closed 4. Bugs opened (2) Bug #1859759 (keystone:Undecided): Keystone is unable to remove role-assignment for deleted LDAP users - Opened by Eigil Obrestad https://bugs.launchpad.net/keystone/+bug/1859759 Bug #1859844 (keystone:Undecided): Impossible to rename the Default domain id to the string 'default.' - Opened by Marcelo Subtil Marcal https://bugs.launchpad.net/keystone/+bug/1859844 Bugs closed (4) Bug #1833207 (keystoneauth:Undecided) https://bugs.launchpad.net/keystoneauth/+bug/1833207 Bug #1858189 (keystoneauth:Undecided) https://bugs.launchpad.net/keystoneauth/+bug/1858189 Bug #1857086 (keystone:Won't Fix) https://bugs.launchpad.net/keystone/+bug/1857086 Bug #1859844 (keystone:Invalid) https://bugs.launchpad.net/keystone/+bug/1859844 ## Milestone Outlook https://releases.openstack.org/ussuri/schedule.html Reminder for Spec freeze as it is on the week of 10 Feburary. ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1 0

[all] cirros-cloud.net is down
by Radosław Piliszek 16 Jan '20

16 Jan '20

Hi, Folks! If your CI jobs depend on downloaded cirros image, then be aware that cirros links are seemingly permanently down atm. I reported [1] due to lack of a better place (that I know of). [1] https://github.com/cirros-dev/cirros/issues/12 -yoctozepto

2 2

[tc] January meeting agenda
by Jean-Philippe Evrard 16 Jan '20

16 Jan '20

Hello everyone, Our next meeting is happening next week Thursday (the 16th), and the agenda is, as usual, on the wiki! Here is a primer of the agenda for this month: - report on large scale sig -- how does this fly and how/what are the action items. - report on the vision reflection update - report on the analysis of the survey - report on the convo for Telemetry with Catalyst -- where are we now? What are the next steps (Gnocchi fork)? - report on multi-arch SIG - report on infra liaison and static hosting -- check if there is progress - report on stable branch policy work. - report on the oslo metrics project -- has code appeared since last convo? - report on the community goals for U and V, py2 drop, and goal select process schedule. - report on release naming - report on the ideas repo See you there! Regards, Jean-Philippe Evrard (evrardjp)

1 1