December 2019 - openstack-discuss

[kolla][cinder][glance][ceph] Corrupt image while downloading from Ceph
by Giuseppe Sannino 03 Dec '19

03 Dec '19

Hi community, need your help. *>>> Background <<<* I'm using kolla-ansible 8.0.0 to deploy a 1+3 "Stein" cluster. Ceph is used as backend. The configuration is a bit peculiar. The control runs on a VM hosted in a separate network compared to the one where the baremetal servers hosting the OS Compute services are. On the Compute Hosts, we have the following services: glance_api neutron_metadata_agent neutron_l3_agent neutron_dhcp_agent neutron_openvswitch_agent openvswitch_vswitchd openvswitch_db nova_compute nova_libvirt nova_ssh cinder_backup cinder_volume chrony cron kolla_toolbox fluentd Services APIs and Authentication run on the controller. In a standard "lab configuration" everything works fine. *>>> Fault Scenario <<<* We are trying to verify possible issues (and the way to work around them) in case latency between Controller and Compuite increases. And we have found one quite fast. Basically, if you try to create a volume from a RAW image (stored in Ceph) it will fail. >From glance-api.log on the controller: 2019-12-03 16:00:11.840 27 INFO eventlet.wsgi.server [req-225aae45-ad93-40f5-835d-027f93e3307d 615252134b844dbeb7acc34219e431e6 0049baebd0f742de915b11ec18509803 - default default] Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py", line 572, in handle_one_response write(b''.join(towrite)) File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py", line 518, in write wfile.writelines(towrite) File "/usr/lib64/python2.7/socket.py", line 334, in writelines self.flush() File "/usr/lib64/python2.7/socket.py", line 303, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 401, in sendall tail = self.send(data, flags) File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 395, in send return self._send_loop(self.fd.send, data, flags) File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 382, in _send_loop return send_method(data, *args) error: [Errno 104] Connection reset by peer >From the cinder-volume.log on the computes: : 019-12-03 16:00:15.932 34 ERROR oslo_messaging.rpc.server None, None) 2019-12-03 16:00:15.932 34 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/image_utils.py", line 410, in fetch 2019-12-03 16:00:15.932 34 ERROR oslo_messaging.rpc.server reason=reason) 2019-12-03 16:00:15.932 34 ERROR oslo_messaging.rpc.server ImageDownloadFailed: Failed to download image 6e7bb902-917e-4c9e-ba9f-3ee811a2502a, reason: IOError: 32 Corrupt image download. Hash was 88b062103e34c9824d7172afaa9a80befd00e1bef86d16a362572f01bd887a0551c188e98526eecdeedca262d3364175d384352c10d203bdb6a5b87b0593f231 expected adc29d5ce6129337e1e9bf00cc3f0798682c021c6f1a0aab46213438a6de8c6b027180389aa21196e7f708214815221a9a0c6029a96badafefca624bf58e4bff *>>> Troubleshooting <<<* At a first glance it seems a problem related to the size of the image. We have tried with: Cirros Raw (39MB) => It works Ubuntu18 QCOW2 (328MB) => It works Ubuntu18 Raw (2.2GB) => IT FAILS !!!! Any suggestion about where to address our effort? Many thanks in advance BR /Giuseppe

1 0

Re: [External] Re: [openstack-community] how do I revert a volume snapshot?
by Amy Marrich 03 Dec '19

03 Dec '19

John, Sorry I misread what you were trying to do. I thought you wanted to create a new instance with the snapshot not revert to it. I've included the link for how to revert volumes. https://docs.openstack.org/cinder/pike/admin/blockstorage-volume-backups.ht… I'm also forwarding this email to the OpenStack Discuss list which is the appropriate list for dev and ops issues. Thanks, Amy (spotz) On Tue, Dec 3, 2019 at 9:29 AM Ratliff, John <jdratlif(a)iu.edu> wrote: > Thanks, but that feels like a really complicated process to rollback. It > seems I would need to delete my old machine and volume, make a volume from > the snapshot, create a new one with the same port, a new volume and specify > the cloud-init script. > > > > Is that really the best way to revert to a snapshot? I don’t have a lot of > experience with openstack, and the documentation and scope is quite large, > but I’ve used vmware, libvirt with qemu, hyper-v, and virtualbox, and they > all had a fairly simple one-step process to revert a snapshot. I feel like > I’m missing something easy. > > > > Thanks. > > > > --John > > > > *From:* Amy Marrich <amy(a)demarco.com> > *Sent:* Monday, December 2, 2019 1:49 PM > *To:* Ratliff, John <jdratlif(a)iu.edu> > *Cc:* community(a)lists.openstack.org > *Subject:* [External] Re: [openstack-community] how do I revert a volume > snapshot? > > > > This message was sent from a non-IU address. Please exercise caution when > clicking links or opening attachments from external sources. > > > > You should be able to create a new instance using that snapshot as the > image. > > > > Amy (spotz) > > > > On Mon, Dec 2, 2019 at 11:27 AM Ratliff, John <jdratlif(a)iu.edu> wrote: > > I am trying to create an instance that I can revert to a snapshot. I > created the snapshot with openstack volume snapshot create, but I don’t see > a way to revert the instance to that image. > > > > What can I do to make use of the snapshot? > > > > Thanks. > > > > --John Ratliff > > > > > > > > _______________________________________________ > Community mailing list > Community(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/community > >

1 0

[ops] ops meetups team meeting 2019-12-3
by Chris Morgan 03 Dec '19

03 Dec '19

Minutes from todays meeting on #openstack-operators Minutes: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… Minutes (text): http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… Log: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… London meetup links: https://go.bloomberg.com/attend/invite/openstack-operators-meetup-london202… https://etherpad.openstack.org/p/LON-2020-OPS-TOPICS South Korea meetup proposal: https://etherpad.openstack.org/p/ops-meetup-2nd-korea-2020 Event notification twitter account: https://twitter.com/osopsmeetup Chris Morgan - on behalf of the OpenStack Ops Meetups team -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[neutron] Bug Deputy report 2019-11-25 - 2019-12-02
by Nate Johnston 03 Dec '19

03 Dec '19

Here are my notes as Neutron bug deputy for last week. It was a pretty slow week. Critical: - "Old conjunction left after sg update" * URL: https://bugs.launchpad.net/neutron/+bug/1854131 * Status: Fix merged (yang-li) - "py36 unit test cases fails" * URL: https://bugs.launchpad.net/bugs/1854051 * Gate-Failure * Status: Unassigned, may be a typing issue, will bring it up in Neutron CI meeting. High: - "Neutron fails to create bandwidth providers if CONF.host is set" * URL: https://bugs.launchpad.net/neutron/+bug/1853840 * Status: In progress, assigned to rubasov - "[Functional tests] Timeout exception in list_namespace_pids" * URL: https://bugs.launchpad.net/bugs/1854462 * Gate-Failure * Status: Unassigned, will bring it up in Neutron CI meeting. Medium: - "The /v2.0/ports/{port_id}/bindings APIs are not documented" * URL: https://bugs.launchpad.net/bugs/1853873 * Status: Assigned to mlavalle Low: - "Avoid raising NetworkInterfaceNotFound exception in DHCP agent logs" * URL: https://bugs.launchpad.net/bugs/1854723 * Status: In progress, assigned to ralonsoh - "DhcpLocalProcess "_enable" method should not call 'restart' --> 'enable'" * URL: https://bugs.launchpad.net/neutron/+bug/1854940 * Status: Assigned to ralonsoh Invalid: - "minor versions 14.0.2 & 14.0.3 are not compatible in dvr-ha" * URL: https://bugs.launchpad.net/bugs/1854050 * Status: agent running higher version than neutron server not supported Thanks, Nate

1 0

Re: [openstack-community] Openstack FW rules
by Amy Marrich 03 Dec '19

03 Dec '19

Hi Lior, Forwarding this to the OpenStack discuss list where someone might be able to help you. Thanks, Amy (spotz) On Tue, Dec 3, 2019 at 7:41 AM Lior Fellus <felluslior(a)gmail.com> wrote: > Hi, > i am using OpenStack "Mitaka" and we have 3 controllers and 3 computes on > our environment and all managed by Fuel server. > recently i implemented a Veeam backup Server in our environment and in > order to backup the computes and the controllers i need to open ssh to them > directly because Veeam not support ssh tunnel via Fuel server. > can anyone explain me how ? > > Regards, > _______________________________________________ > Community mailing list > Community(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/community >

1 0

VMs auto start
by Alfredo De Luca 03 Dec '19

03 Dec '19

Hi all. Just a quick one. I can't find the settings for VMs auto start after a computer node reboots. Can you point me to the right docs? Which one is it? And you need to add it on nova scheduler right? Cheers -- *Alfredo*

2 2

[gnocchi][telemetry][ceilometer][cloudkitty] Gnocchi unmaintained
by Radosław Piliszek 03 Dec '19

03 Dec '19

Hello Folks, It looks like gnocchi is "officially" marked as unmaintained: https://github.com/gnocchixyz/gnocchi/issues/1049 Has there been any discussion regarding how it affects OpenStack projects? And/or are there any plans to amend this situation? -yoctozepto

15 28

[requirements][os-ken][neutron] update of tinyrpc from 1.0.3 to 1.0.4 breaks neutron-grenade
by Matthew Thode 03 Dec '19

03 Dec '19

2 2

Magnum: coe cluster can not be deleted (DELETE_FAILED)
by Pawel Konczalski 03 Dec '19

03 Dec '19

Hello, i try to delete a broken magnum (v7.1.1.dev21) cluster but neither the cluster or the stack entries can be deleted. Any idea how to force / fix this? BR Pawel openstack coe cluster list | grep DELETE_FAILED openstack coe cluster delete ae5c347c-3ba8-4d39-9125-0bdadef253b4 openstack coe cluster show ae5c347c-3ba8-4d39-9125-0bdadef253b4 | status | DELETE_FAILED ... |--- | status_reason | Resource DELETE failed: JSONDecodeError: resources.kube_masters.resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0) ... | faults | {'1': 'JSONDecodeError: resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0)', 'kube_masters': 'JSONDecodeError: resources.kube_masters.resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0)', '0': 'JSONDecodeError: resources[0].resources.api_pool_member: Expecting value: line 1 column 1 (char 0)', 'api_pool_member': 'JSONDecodeError: resources.api_pool_member: Expecting value: line 1 column 1 (char 0)', 'etcd_pool_member': 'JSONDecodeError: resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0)'} | openstack stack list | grep DELETE_FAILED openstack stack delete 2a907fc4-7b40-43f6-9254-d4084bd056b8 t Payload DELETE: ResourceGroup "kube_masters" [85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec] Stack "slu-k8s-cluster2-6xiqrpfe7jd6" [2a907fc4-7b40-43f6-9254-d4084bd056b8] 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource Traceback (most recent call last): 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py", line 924, in _action_recorder 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource yield 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py", line 2034, in delete 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource *action_args) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource step = next(subtask) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resource.py", line 986, in action_handler_task 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource done = check(handler_data) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 596, in check_delete_complete 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource return self._check_status_complete(self.DELETE) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 463, in _check_status_complete 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource action=action) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource ResourceFailure: JSONDecodeError: resources.kube_masters.resources[1].resources.etcd_pool_member: Expecting value: line 1 column 1 (char 0) 2019-12-02 12:43:08.887 22 ERROR heat.engine.resource # openstack stack list --nested | grep 7390a9b1d4be4d75b4bd08ab8107e4ff | 77848f9e-ff0c-4f7a-9fd7-8ed9e979b998 | test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd-1-tpqatosey33u | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:28:04Z | 2019-12-02T15:30:53Z | 85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec | | 85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec | test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:28:03Z | 2019-12-02T15:30:53Z | 2a907fc4-7b40-43f6-9254-d4084bd056b8 | | 4c91c53b-42f4-41ca-a97c-49d9261069d6 | test-k8s-cluster2-6xiqrpfe7jd6-kube_masters-mlvsrucewmsd-0-hm2ucynq6hkc | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:28:03Z | 2019-12-02T15:30:54Z | 85e1c6be-bdc8-41a3-ba0f-5ca65cdea8ec | | 2a907fc4-7b40-43f6-9254-d4084bd056b8 | test-k8s-cluster2-6xiqrpfe7jd6 | 7390a9b1d4be4d75b4bd08ab8107e4ff | DELETE_FAILED | 2019-05-13T15:24:45Z | 2019-12-02T15:30:53Z | None |

2 1

[all] Shanghai PTG Feedback
by Kendall Nelson 03 Dec '19

03 Dec '19

Hello Everyone! Now that we've had a few weeks to catch up after the summit and marinate on everything we accomplished together there, we wanted to take a sec to see if anyone had feedback. Normally we would have held a lunchtime feedback session, but due to venue restrictions, we didn't have a good place to do things like normal so we created the etherpad[1] and are distributing it this way :) Hope to hear everyone's feedback! Thanks! -the Kendalls (diablo_rojo & wendallkaters) [1]https://etherpad.openstack.org/p/PVG-PTG-feedback

1 0