- openstack-discuss - lists.openstack.org

Swedish University Engagement Program
by Kendall Nelson 05 Dec '19

05 Dec '19

Hello Everyone! During the PTG, I talked to Kim Hindhart (along with Ildiko Vancsa, Sean McGinnis, and Victoria Martinez de la Cruz) about a program he was working on with several universities in Sweden (Blekinge Institute of Technology, Uppsala University, and KTH Royal Institute of Technology) to get students involved in open source, specifically involved in OpenStack. While this program is still getting off the ground, with its first round to start in this coming February with students working on Horizon (led by Kim), we are hoping to collect work for the second round that will be more student driven. The only disclaimer is that items suggested might not get picked up until the fall semester in September. I am collecting low-hanging-fruit here on this etherpad[1].Hopefully you have a lot of things tagged as such on Launchpad/StoryBoard already and can easily add those links to the etherpad. Can't wait to see what we can come up with! -Kendall (diablo_rojo) [1] https://etherpad.openstack.org/p/swedish-university-program-work

1 0

[tripleo][ppc64le] Unexpected Return Code from TripleO Container Build Playbooks on ppc64le
by Adam Kimball 04 Dec '19

04 Dec '19

Hello all, Mike Turek and I have been working to get a ppc64le job to build tripleo containers for quite some time now. A few weeks back, we stumbled into a strange issue where the containers build job was issuing a return code 1, despite all containers successfully building. This has largely blocked our ability to push containers up to docker hub or report success through the delorean API. We've been pushing a number of patches to expand the logging of both our job, and the tripleo playbooks themselves, to get a better idea of what's going on. The most recent was a patch to show which containers have python futures that have been throwing exceptions, and why. [1] The end result of this seems to be that a number of jobs are reporting as incomplete. An example of this can be seen at timestamp 20:16:07 of an example build log. [2] However, upon checking the list of successfully built containers [3], or the RDO registry itself [4], one can see that the containers producing job not complete errors have actually built, and are being uploaded. The error log generated by the tripleo playbooks is also empty. [5] At this point, we're wondering what the path forward is. It seems like the issue stems from some unintended behavior in the tripleo playbooks themselves, not anything from our job. We're trying to figure out if this behavior is something that should be preventing us from reporting successful builds, and if so, how it can be fixed. Thanks, Adam Kimball [1] - https://review.opendev.org/#/c/695723/ [2] - https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master… [3] - https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master… [4] - https://console.registry.rdoproject.org/registry [5] - https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master…

2 1

[cinder] Ussuri Shanghai and Virtual PTG update
by Brian Rosmaita 04 Dec '19

04 Dec '19

The PTG summary document has been updated with a quick synopsis of what we discussed at the Virtual PTG last week: https://wiki.openstack.org/wiki/CinderUssuriPTGSummary All sessions (both Shanghai and Virtual) were recorded. The above page contains links to the recordings, which you can also find on the Cinder YouTube channel. Please look through the summary document to remind yourself about any actions you volunteered for. Additionally, there's an etherpad of unclaimed action items if you're looking for something to do: https://etherpad.openstack.org/p/cinder-ussuri-ptg-actions Thanks to everyone who participated in the PTG events. cheers, brian

1 0

[manila] Remote-only PTG poll
by Goutham Pacha Ravi 04 Dec '19

04 Dec '19

Hey Zorillas* and interested Stackers, (* manila contributors in the wild) A part of our community will be attending the Summit+Forum+PTG in Shanghai in a couple of weeks. However, we haven't figured out a way to enable remote participation during our PTG (Wednesday, 6th Nov 2019 between 9:00 AM and 4:30 PM) there. So, as discussed during our weekly meeting, I'd like to take a poll of when you would like to gather for a remote PTG that would supplement the gathering in Shanghai. Please take this poll and indicate your availability: https://xoyondo.com/dp/qQbsoHtNhi4DCki The prospective date for this meeting is between 2nd Dec and 13th Dec (Ussuri-1 Milestone is on 12th Dec). The meeting time will be 15:00 UTC to 22:00 UTC. This time may not work for our contributors in the Far East (APAC), but I'm hoping we'll address them primarily at the Shanghai get together, and relay their thoughts here. We will also be recording the entire meeting and will share notes on these lists. Thanks, Goutham

1 1

DHCP timeout when creating instances for specific tenants
by Grant Morley 04 Dec '19

04 Dec '19

Hi all, I wonder if anyone can help shed any light on an odd issue we are seeing with only a couple of specific tenants. Basically if they launch an instance they are taking about 5 minutes to launch rather than our usual 30 second or so launch. We are seeing the following on the instance logs: https://pastebin.com/hDstsd8G Weirdly it only seems to be happening for 1 or 2 new tenants. I have tested this on our personal account and a few other customers have tested and their instances launch really quickly as expected. Is there anything specific during the tenant creation that can cause this issue? Or are there any logs in nova / neutron I should be looking out for that might shed some light? I haven't seen anything that is obvious. Any help would be much appreciated as we are a little stumped at the moment. Many thanks, -- Grant Morley Cloud Lead, Civo Ltd www.civo.com <https://www.civo.com/>| Signup for an account! <https://www.civo.com/signup>

1 0

[neutron] api performance at scale
by Erik Olof Gunnar Andersson 04 Dec '19

04 Dec '19

Is there a SIG or similar discussion on neutron performance at scale? For us nova used to be the biggest concern, but a lot of work has been done and nova now performers great. Instead we are having issues to get Neutron to perform at scale. Obvious calls like security groups are performing really poorly, and nova-compute defaults for refreshing the network cache on computes causes massive issues with Neutron. Best Regards, Erik Olof Gunnar Andersson

3 5

[horizon] [plugins] Supported Django version for horizon in Ussuri
by vishal manchanda 04 Dec '19

04 Dec '19

Hello Everyone, As discussed at this weekly meeting[1] and Shanghai PTG[2], Horizon team is going to bump default version of Django to the next LTS release which is 2.2. Django 2.2 support is already added in Horizon and all its plugins [3]. Thanks to all for taking this at the priority and quick approvals on the patches. After Switching the default version of Django to 2.2(latest LTS), We will drop Django 1.11 in the next step(as its extended supports ending soon [4]). Also, Horizon usually supports LTS version of Django[5]. Please find below plan for coming weeks: Step 1: Switch the default version to Django 2.2 in upper-constraints till milestone-1. Additionally, Due to these changes, there might be a possibility of potential risks. Horizon and horizon plugins already support Django 2.2 as unit tests, but they are not tested in the real deployments with Django 2.2. So After switching the default version, if anything breaks we will keep working on fixing those issues. Step 2: Drop Django 1.11 during milestone-2. In case of any issue, please feel free to reach-out to Horizon channel #openstack-horizon. [1] http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstac… [2] https://etherpad.openstack.org/p/horizon-u-ptg#45 [3] https://review.opendev.org/#/q/topic:django22+(status:open+OR+status:merged) [4] https://www.djangoproject.com/weblog/2015/jun/25/roadmap/ [5] https://docs.openstack.org/horizon/latest/contributor/policy.html#django-su… [6] https://docs.djangoproject.com/en/2.2/releases/2.2/ cheers, Vishal Manchanda

1 0

[glance] [glance_store] requirements-check job fails
by Naohiro Sameshima（鮫島直洋）(Group) 04 Dec '19

04 Dec '19

Hi, I'm Naohiro. I proposed a blueprint [0] (revive s3 driver) in Shanghai PTG and then create a patch [1]. So, I submit a patch in gerrit code review, but unfortunately one of zuul job (requirements-check) failed [2]. I think it seems that doc/requirements.txt is wrong. (Not included in my patch) Error log is below. ~~~~~~~~~~~~~~~~~~~~ Validating doc/requirements.txt WARNING: possible mismatch found for package "sphinx" Attribute "markers" does not match "" does not match "python_version>='3.4'" Requirement(package='sphinx', location='', specifiers='!=1.6.6,!=1.6.7,>=1.6.2', markers='', comment='# BSD', extras=frozenset()) Requirement(package='sphinx', location='', specifiers='!=1.6.6,!=1.6.7,!=2.1.0', markers="python_version>='3.4'", comment='# BSD', extras=frozenset()) WARNING: possible mismatch found for package "sphinx" Attribute "markers" does not match "" does not match "python_version=='2.7'" Requirement(package='sphinx', location='', specifiers='!=1.6.6,!=1.6.7,>=1.6.2', markers='', comment='# BSD', extras=frozenset()) Requirement(package='sphinx', location='', specifiers='!=1.6.6,!=1.6.7,<2.0.0', markers="python_version=='2.7'", comment='# BSD', extras=frozenset()) ERROR: Could not find a global requirements entry to match package sphinx. If the package is already included in the global list, the name or platform markers there may not match the local settings. ~~~~~~~~~~~~~~~~~~~~ Or is above job failing because of my patch? Thank you. Naohiro Sameshima [0]: https://review.opendev.org/#/c/687390/ [1]: https://review.opendev.org/#/c/695844/ [2]: https://zuul.opendev.org/t/openstack/build/4aef8abff3374613ab413aca0ab65381 This email and all contents are subject to the following disclaimer: https://hello.global.ntt/en-us/email-disclaimer

2 2

[tc] Meetings: Agenda for december 2019 and new date for January 2020
by Jean-Philippe Evrard 04 Dec '19

04 Dec '19

Hello everyone, Our next meeting is happening tomorrow 5 december, and the agenda is... still on the wiki. If you want it's also a little below [1]. I want to also highlight that the January 2020 meeting is happening on Thursday 16 instead of the first Thursday as usual. Regards, Jean-Philippe Evrard (evrardjp) [1]: Agenda for this meeting: Follow up on past action items: ricolin: (refresh) Follow up with SIG chairs about guidelines https://etherpad.openstack.org/p/SIGs-guideline ttx: (refresh) contact interested parties in a new 'large scale' sig (help with mnaser, jroll reaching out to verizon media) New initiatives and/or report on previous initiatives: gmann: report on the community goals for U and V, py2 drop, and goal select process schedule. anyone to help gmann? mnaser: sync up with swift team on python3 migration ricolin: report on multi-arch sig and other sigs ttx: Technical vision reflection update mnaser: Summary of the maintain issue with Telemetry mnaser: report oslo metrics project jungleboyj: update on the blog post about the analysis of the Foundation user survey mugsie: release naming progress mnaser: report on the infra liaison -- what's happening with the static hosting ttx: merge tc/uc update evrardjp: concepts repo addendums: gmann: define goal select process schedule -- is the current state clear? evrardjp: Patches pending mnaser: Summary of the stable branch policy discussion since the summit

1 0

[scientific-sig] IRC meeting today - unified limits for mixed baremetal/virt
by Stig Telfer 04 Dec '19

04 Dec '19

Hi All - We have a Scientific SIG IRC meeting at 1100 UTC in channel #openstack-meeting. Everyone is welcome. Today’s agenda is available here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_December_4th_2019 We’d like to discuss an update on unified limits, and how that will enable useful capabilities such as enforcing quotas across mixed environments - virtual, baremetal, GPUs, etc. Cheers, Stig (oneswig)

1 0

[Watcher] No IRC meeting today
by licanwei 04 Dec '19

04 Dec '19

1 0

Add functionality to Swift
by Anmar Salih 04 Dec '19

04 Dec '19

Hey, I have four containers in Openstack Swift. One of the containers called "Cont1", and I am using the web browser to upload and download objects to/from Cont1. I need to add a script that runs each time someone downloads an object from the container Cont1. Let's say I want Swift to execute a small script every time someone clicks on the Download button. Wondering if that possible? Where I should add the code. I cloned Swift Stein git on my local machine but not sure where to add my script. Take a look at the attached image, to the very right there is a download button for object1. So when someone clicks on that button, I want to execute a small script, then after executing the script, Swift will start to download object1. https://imgur.com/cCZIJqR Note: I need to add functionality to Swift, so when someone requests an object download, swift will take an action based on the container name. Best Regards. Anmar Salih

1 0

Thread on OpenDev Independence and Governance
by Clark Boylan 03 Dec '19

03 Dec '19

Hello, I wanted everyone to be aware of this thread [0] at openstack-infra(a)lists.openstack.org that I've started to formalize OpenDev's future independence and governance. Currently the infra team is formally an OpenStack project, but we think that we'll be better able to serve our non OpenStack users if the OpenDev aspects of this team become independent with its own governance. We are more than happy to hear feedback so please read this thread and respond if the topic interests you. Note, I'm using this pointer thread in an effort to avoid cross posting and losing feedback. It would be great if people can respond to the original on openstack-infra@lists. [0] http://lists.openstack.org/pipermail/openstack-infra/2019-December/006537.h… Thank you, Clark

1 0

[neutron][nova] Rootwrap daemon and privsep
by Arnaud Morin 03 Dec '19

03 Dec '19

Hey, If I believe both the nova and neutron documentation: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.us… https://docs.openstack.org/neutron/latest/configuration/neutron.html#agent.… At scale, we are supposed to enable the rootwrap-daemon option. I have issues enabling that, but before going further on my issue, I'd like to understand the difference with privsep daemon. Is privsep a new daemon which is supposed to replace the rootwrap one? Is privsep being launch after rootwrap? IS privsep enabled by default, so I should not care about rootwrap at all? I'd like to understand more about that. Everything I try to find on popular search engine seems outdated, so if someone could give me a hand on finding the good page to understand that, I'l love it :p Cheers, -- Arnaud Morin

3 7

[Thirdparty CI] Thirdparty CI Site is down
by Rajini.Karthik＠Dell.com 03 Dec '19

03 Dec '19

The thirdparty CI site is down http://ciwatch.mmedvede.net/project?project=ironic&time=7+days, no results are displayed. Anyone has any insights? Thanks Rajini

2 1

[cinder] weekly meeting change reminder
by Brian Rosmaita 03 Dec '19

03 Dec '19

This is a quick reminder that this week's Cinder meeting will be held as usual on Wednesday, but: new time: 1400 UTC new channel: #openstack-meeting-4 See you there! brian

1 0

[kolla][cinder][glance][ceph] Corrupt image while downloading from Ceph
by Giuseppe Sannino 03 Dec '19

03 Dec '19

Hi community, need your help. *>>> Background <<<* I'm using kolla-ansible 8.0.0 to deploy a 1+3 "Stein" cluster. Ceph is used as backend. The configuration is a bit peculiar. The control runs on a VM hosted in a separate network compared to the one where the baremetal servers hosting the OS Compute services are. On the Compute Hosts, we have the following services: glance_api neutron_metadata_agent neutron_l3_agent neutron_dhcp_agent neutron_openvswitch_agent openvswitch_vswitchd openvswitch_db nova_compute nova_libvirt nova_ssh cinder_backup cinder_volume chrony cron kolla_toolbox fluentd Services APIs and Authentication run on the controller. In a standard "lab configuration" everything works fine. *>>> Fault Scenario <<<* We are trying to verify possible issues (and the way to work around them) in case latency between Controller and Compuite increases. And we have found one quite fast. Basically, if you try to create a volume from a RAW image (stored in Ceph) it will fail. >From glance-api.log on the controller: 2019-12-03 16:00:11.840 27 INFO eventlet.wsgi.server [req-225aae45-ad93-40f5-835d-027f93e3307d 615252134b844dbeb7acc34219e431e6 0049baebd0f742de915b11ec18509803 - default default] Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py", line 572, in handle_one_response write(b''.join(towrite)) File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py", line 518, in write wfile.writelines(towrite) File "/usr/lib64/python2.7/socket.py", line 334, in writelines self.flush() File "/usr/lib64/python2.7/socket.py", line 303, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 401, in sendall tail = self.send(data, flags) File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 395, in send return self._send_loop(self.fd.send, data, flags) File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 382, in _send_loop return send_method(data, *args) error: [Errno 104] Connection reset by peer >From the cinder-volume.log on the computes: : 019-12-03 16:00:15.932 34 ERROR oslo_messaging.rpc.server None, None) 2019-12-03 16:00:15.932 34 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/image_utils.py", line 410, in fetch 2019-12-03 16:00:15.932 34 ERROR oslo_messaging.rpc.server reason=reason) 2019-12-03 16:00:15.932 34 ERROR oslo_messaging.rpc.server ImageDownloadFailed: Failed to download image 6e7bb902-917e-4c9e-ba9f-3ee811a2502a, reason: IOError: 32 Corrupt image download. Hash was 88b062103e34c9824d7172afaa9a80befd00e1bef86d16a362572f01bd887a0551c188e98526eecdeedca262d3364175d384352c10d203bdb6a5b87b0593f231 expected adc29d5ce6129337e1e9bf00cc3f0798682c021c6f1a0aab46213438a6de8c6b027180389aa21196e7f708214815221a9a0c6029a96badafefca624bf58e4bff *>>> Troubleshooting <<<* At a first glance it seems a problem related to the size of the image. We have tried with: Cirros Raw (39MB) => It works Ubuntu18 QCOW2 (328MB) => It works Ubuntu18 Raw (2.2GB) => IT FAILS !!!! Any suggestion about where to address our effort? Many thanks in advance BR /Giuseppe

1 0

Re: [External] Re: [openstack-community] how do I revert a volume snapshot?
by Amy Marrich 03 Dec '19

03 Dec '19

John, Sorry I misread what you were trying to do. I thought you wanted to create a new instance with the snapshot not revert to it. I've included the link for how to revert volumes. https://docs.openstack.org/cinder/pike/admin/blockstorage-volume-backups.ht… I'm also forwarding this email to the OpenStack Discuss list which is the appropriate list for dev and ops issues. Thanks, Amy (spotz) On Tue, Dec 3, 2019 at 9:29 AM Ratliff, John <jdratlif(a)iu.edu> wrote: > Thanks, but that feels like a really complicated process to rollback. It > seems I would need to delete my old machine and volume, make a volume from > the snapshot, create a new one with the same port, a new volume and specify > the cloud-init script. > > > > Is that really the best way to revert to a snapshot? I don’t have a lot of > experience with openstack, and the documentation and scope is quite large, > but I’ve used vmware, libvirt with qemu, hyper-v, and virtualbox, and they > all had a fairly simple one-step process to revert a snapshot. I feel like > I’m missing something easy. > > > > Thanks. > > > > --John > > > > *From:* Amy Marrich <amy(a)demarco.com> > *Sent:* Monday, December 2, 2019 1:49 PM > *To:* Ratliff, John <jdratlif(a)iu.edu> > *Cc:* community(a)lists.openstack.org > *Subject:* [External] Re: [openstack-community] how do I revert a volume > snapshot? > > > > This message was sent from a non-IU address. Please exercise caution when > clicking links or opening attachments from external sources. > > > > You should be able to create a new instance using that snapshot as the > image. > > > > Amy (spotz) > > > > On Mon, Dec 2, 2019 at 11:27 AM Ratliff, John <jdratlif(a)iu.edu> wrote: > > I am trying to create an instance that I can revert to a snapshot. I > created the snapshot with openstack volume snapshot create, but I don’t see > a way to revert the instance to that image. > > > > What can I do to make use of the snapshot? > > > > Thanks. > > > > --John Ratliff > > > > > > > > _______________________________________________ > Community mailing list > Community(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/community > >

1 0

[ops] ops meetups team meeting 2019-12-3
by Chris Morgan 03 Dec '19

03 Dec '19

Minutes from todays meeting on #openstack-operators Minutes: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… Minutes (text): http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… Log: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… London meetup links: https://go.bloomberg.com/attend/invite/openstack-operators-meetup-london202… https://etherpad.openstack.org/p/LON-2020-OPS-TOPICS South Korea meetup proposal: https://etherpad.openstack.org/p/ops-meetup-2nd-korea-2020 Event notification twitter account: https://twitter.com/osopsmeetup Chris Morgan - on behalf of the OpenStack Ops Meetups team -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[neutron] Bug Deputy report 2019-11-25 - 2019-12-02
by Nate Johnston 03 Dec '19

03 Dec '19

Here are my notes as Neutron bug deputy for last week. It was a pretty slow week. Critical: - "Old conjunction left after sg update" * URL: https://bugs.launchpad.net/neutron/+bug/1854131 * Status: Fix merged (yang-li) - "py36 unit test cases fails" * URL: https://bugs.launchpad.net/bugs/1854051 * Gate-Failure * Status: Unassigned, may be a typing issue, will bring it up in Neutron CI meeting. High: - "Neutron fails to create bandwidth providers if CONF.host is set" * URL: https://bugs.launchpad.net/neutron/+bug/1853840 * Status: In progress, assigned to rubasov - "[Functional tests] Timeout exception in list_namespace_pids" * URL: https://bugs.launchpad.net/bugs/1854462 * Gate-Failure * Status: Unassigned, will bring it up in Neutron CI meeting. Medium: - "The /v2.0/ports/{port_id}/bindings APIs are not documented" * URL: https://bugs.launchpad.net/bugs/1853873 * Status: Assigned to mlavalle Low: - "Avoid raising NetworkInterfaceNotFound exception in DHCP agent logs" * URL: https://bugs.launchpad.net/bugs/1854723 * Status: In progress, assigned to ralonsoh - "DhcpLocalProcess "_enable" method should not call 'restart' --> 'enable'" * URL: https://bugs.launchpad.net/neutron/+bug/1854940 * Status: Assigned to ralonsoh Invalid: - "minor versions 14.0.2 & 14.0.3 are not compatible in dvr-ha" * URL: https://bugs.launchpad.net/bugs/1854050 * Status: agent running higher version than neutron server not supported Thanks, Nate

1 0

Re: [openstack-community] Openstack FW rules
by Amy Marrich 03 Dec '19

03 Dec '19

Hi Lior, Forwarding this to the OpenStack discuss list where someone might be able to help you. Thanks, Amy (spotz) On Tue, Dec 3, 2019 at 7:41 AM Lior Fellus <felluslior(a)gmail.com> wrote: > Hi, > i am using OpenStack "Mitaka" and we have 3 controllers and 3 computes on > our environment and all managed by Fuel server. > recently i implemented a Veeam backup Server in our environment and in > order to backup the computes and the controllers i need to open ssh to them > directly because Veeam not support ssh tunnel via Fuel server. > can anyone explain me how ? > > Regards, > _______________________________________________ > Community mailing list > Community(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/community >

1 0

VMs auto start
by Alfredo De Luca 03 Dec '19

03 Dec '19

Hi all. Just a quick one. I can't find the settings for VMs auto start after a computer node reboots. Can you point me to the right docs? Which one is it? And you need to add it on nova scheduler right? Cheers -- *Alfredo*

2 2

[gnocchi][telemetry][ceilometer][cloudkitty] Gnocchi unmaintained
by Radosław Piliszek 03 Dec '19

03 Dec '19

Hello Folks, It looks like gnocchi is "officially" marked as unmaintained: https://github.com/gnocchixyz/gnocchi/issues/1049 Has there been any discussion regarding how it affects OpenStack projects? And/or are there any plans to amend this situation? -yoctozepto

15 28

[requirements][os-ken][neutron] update of tinyrpc from 1.0.3 to 1.0.4 breaks neutron-grenade
by Matthew Thode 03 Dec '19

03 Dec '19

2 2

Magnum: coe cluster can not be deleted (DELETE_FAILED)
by Pawel Konczalski 03 Dec '19

03 Dec '19

Hello, i try to delete a broken magnum (v7.1.1.dev21) cluster but neither the cluster or the stack entries can be deleted. Any idea how to force / fix this? BR Pawel openstack coe cluster list | grep DELETE_FAILED openstack coe cluster delete ae5c347c-3ba8-4d39-9125-0bdadef253b4 openstack coe cluster show ae5c347c-3ba8-4d39-9125-0bdadef253b4 | status | DELETE_FAILED ... |--- | status_reason | Resource DELETE failed: JSONDecodeError: resources.kube_masters.resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0) ... | faults | {'1': 'JSONDecodeError: resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0)', 'kube_masters': 'JSONDecodeError: resources.kube_masters.resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0)', '0': 'JSONDecodeError: resources[0].resources.api_pool_member: Expecting value: line 1 column 1 (char 0)', 'api_pool_member': 'JSONDecodeError: resources.api_pool_member: Expecting value: line 1 column 1 (char 0)', 'etcd_pool_member': 'JSONDecodeError: resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0)'} | openstack stack list | grep DELETE_FAILED openstack stack delete 2a907fc4-7b40-43f6-9254-d4084bd056b8 t Payload DELETE: ResourceGroup "kube_masters" [85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec] Stack "slu-k8s-cluster2-6xiqrpfe7jd6" [2a907fc4-7b40-43f6-9254-d4084bd056b8] 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource Traceback (most recent call last): 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py", line 924, in _action_recorder 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource yield 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py", line 2034, in delete 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource *action_args) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource step = next(subtask) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py", line 986, in action_handler_task 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource done = check(handler_data) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 596, in check_delete_complete 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource return self._check_status_complete(self.DELETE) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 463, in _check_status_complete 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource action=action) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource ResourceFailure: JSONDecodeError: resources.kube_masters.resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource # openstack stack list --nested | grep 7390a9b1d4be4d75b4bd08ab8107e4ff | 77848f9e-ff0c-4f7a-9fd7-8ed9e979b998 | test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd-1-tpqatosey33u | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:28:04Z | 2019-12-02T15:30:53Z | 85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec | | 85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec | test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:28:03Z | 2019-12-02T15:30:53Z | 2a907fc4-7b40-43f6-9254-d4084bd056b8 | | 4c91c53b-42f4-41ca-a97c-49d9261069d6 | test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd-0-hm2ucynq6hkc | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:28:03Z | 2019-12-02T15:30:54Z | 85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec | | 2a907fc4-7b40-43f6-9254-d4084bd056b8 | test-k8s-cluster2-6xiqrpfe7jd6 | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:24:45Z | 2019-12-02T15:30:53Z | None |

2 1