- openstack-discuss - lists.openstack.org

[magnum] [stein] kubernetes stack does not terminate
by Ignazio Cassano 08 Dec '19

08 Dec '19

hello, I've just installed stein with magnum on centos 7. I am creating a kubernetes cluster but it is running since 22 minutes and I think it will not work because the heat stack is waiting OSSoftwareDeployment completes I presume something is not working on heat with magnum. I ran several stack with Software Deployment without magnum, and they worked fine Thanks Ignazio

2 16

RE: High CPU utilization by systemd process || debian 8
by Md. Farhad Hasan Khan 07 Dec '19

07 Dec '19

Hi, Please help to solve this. Thanks/Rony From: Md. Farhad Hasan Khan [mailto:rony.khan@brilliant.com.bd] Sent: Saturday, December 7, 2019 11:55 AM To: 'OpenStack Discuss' Subject: High CPU utilization by systemd process || debian 8 Hi, We are observing high CPU utilization by systemd process. 100% CPU utilize by systemd process when no user is being logged in to the server but CPU utilization became 0% while any user logged in to the server. What cloud be possible reason behind this issue and solution? Log collected via login to the server via ssh or console: (0% CPU utilization) top -b -n 1 | grep systemd | grep 612 612 root 20 0 3386692 2.051g 4932 T 0.0 6.5 22029:26 systemd Log collected without login to the server : (100% CPU utilization) # ssh user(a)192.168.18.11 <mailto:user@192.168.18.11> top -b -n 1 | grep systemd | grep 612 user(a)192.168.18.11's <mailto:user@192.168.18.11's> password: 612 root 20 0 3386692 2.051g 4932 S 789.2 6.5 22031:31 systemd Please find the server information: lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 8.11 (jessie) Release: 8.11 Codename: jessie uname -r 3.16.0-6-amd64 Thanks & B’Rgds, Rony

1 0

neutron-metadata-agent broken pipe
by Albert Braden 07 Dec '19

07 Dec '19

As our production cluster grows in size we are starting to have trouble with neutron-metadata-agent. After restarting it is happy for a minute and then it complains "2019-12-06 17:54:24.615 664587 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 11, greater than warning threshold: 10. There could be a leak. Increasing threshold to: 20" It increases the threshold a couple of times and then after increasing to 40 we start to see errors: 2019-12-06 17:55:10.119 664578 INFO eventlet.wsgi.server [-] Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 521, in handle_one_response write(b''.join(towrite)) File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 462, in write wfile.flush() File "/usr/lib/python2.7/socket.py", line 307, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 390, in sendall tail = self.send(data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 384, in send return self._send_loop(self.fd.send, data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 371, in _send_loop return send_method(data, *args) error: [Errno 32] Broken pipe It looks like increasing the threshold to 40 fails because it keeps trying: 2019-12-06 17:55:17.452 664597 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 21, greater than warning threshold: 20. There could be a leak. Increasing threshold to: 40 And the errors increase until the log is nothing but errors, and VMs fail to boot. root@us01odc-p02-ctrl1:~# tail -f /var/log/neutron/neutron-metadata-agent.log self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 390, in sendall tail = self.send(data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 384, in send return self._send_loop(self.fd.send, data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 371, in _send_loop return send_method(data, *args) error: [Errno 32] Broken pipe 2019-12-06 17:59:23.664 664597 INFO eventlet.wsgi.server [-] 10.195.73.174,<local> "GET /latest/meta-data/instance-id HTTP/1.0" status: 200 len: 0 time: 62.1517761 2019-12-06 17:59:23.756 664583 INFO eventlet.wsgi.server [-] Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 521, in handle_one_response write(b''.join(towrite)) File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 462, in write wfile.flush() File "/usr/lib/python2.7/socket.py", line 307, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 390, in sendall tail = self.send(data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 384, in send return self._send_loop(self.fd.send, data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 371, in _send_loop return send_method(data, *args) error: [Errno 32] Broken pipe 2019-12-06 17:59:23.757 664583 INFO eventlet.wsgi.server [-] 10.195.65.69,<local> "GET /latest/meta-data/instance-id HTTP/1.0" status: 200 len: 0 time: 63.0419171 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent [-] Unexpected error.: MessagingTimeout: Timed out waiting for a reply to message ID 77551d4cf4394b7b9cdfad68c0be46e8 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent Traceback (most recent call last): 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 89, in __call__ 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent instance_id, tenant_id = self._get_instance_and_tenant_id(req) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 162, in _get_instance_and_tenant_id 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent ports = self._get_ports(remote_address, network_id, router_id) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 155, in _get_ports 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent return self._get_ports_for_remote_address(remote_address, networks) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/common/cache_utils.py", line 116, in __call__ 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent return self.func(target_self, *args, **kwargs) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 137, in _get_ports_for_remote_address 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent ip_address=remote_address) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 106, in _get_ports_from_server 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent return self.plugin_rpc.get_ports(self.context, filters) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 72, in get_ports 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent return cctxt.call(context, 'get_ports', filters=filters) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 173, in call 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent time.sleep(wait) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent self.force_reraise() 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent six.reraise(self.type_, self.value, self.tb) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 150, in call 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent return self._original_context.call(ctxt, method, **kwargs) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 179, in call 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent retry=self.retry) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 133, in _send 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent retry=retry) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent call_monitor_timeout, retry=retry) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent call_monitor_timeout) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent message = self.waiters.get(msg_id, timeout=timeout) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent 'to message ID %s' % msg_id) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent MessagingTimeout: Timed out waiting for a reply to message ID 77551d4cf4394b7b9cdfad68c0be46e8 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent 2019-12-06 17:59:23.778 664595 INFO eventlet.wsgi.server [-] Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 521, in handle_one_response write(b''.join(towrite)) File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 462, in write wfile.flush() File "/usr/lib/python2.7/socket.py", line 307, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 390, in sendall tail = self.send(data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 384, in send return self._send_loop(self.fd.send, data, flags) What's causing this? Are we overloading RMQ?

1 0

[nova][ironic] Changing an owner of a provisioned node
by Matt Riedemann 07 Dec '19

07 Dec '19

The 1.50 microversion [1] in the ironic API added the "owner" field to the node and I'm trying to use that to add some scheduler filtering in nova [2]. It's my understanding that the owner field on a provisioned node (instance_uuid on the node is set) can be changed, but I'm surprised that is allowed. Was that an oversight in developing that feature? The use case for the scheduler filter is baremetal nodes are owned by different (non-admin) projects in a deployment. When a non-admin project creates a baremetal server via nova, nova will filter out nodes that are not owned by the project (based on the node.owner field). If a node isn't owned by any project, only admins can use it. Admins also have access to all nodes regardless of owner. Given that, let's say user 1 from project A creates a server on nova X that is owned by project A (node.owner=A). Then the node.owner is changed to project B. What should happen? Should nova detect that ownership change and stop the node or something? Note that with other resources that can transfer ownership, like volumes, that can only be done when they aren't in use. So why don't we have the same rules for nodes? Assuming we do want to enforce this in the API (a 409 response when trying to change the owner on a provisioned node), how would that be done given this is a problem since 1.50 which was added in Stein? Would a policy rule be added to ironic to determine if someone can change the owner on a provisioned node and if so, what would be the default rule? The same as "baremetal:node:update" (rule:is_admin)? [1] https://docs.openstack.org/ironic/latest/contributor/webapi-version-history… [2] https://blueprints.launchpad.net/nova/+spec/ironic-tenant-filter -- Thanks, Matt

3 3

RE: Rebooted VM disappears when HV lacks memory
by Albert Braden 06 Dec '19

06 Dec '19

Hi Sean, I didn't want to mess with prod configs but shelve/unshelve works. Thanks for your advice! -----Original Message----- From: Sean Mooney <smooney(a)redhat.com> Sent: Friday, December 6, 2019 3:53 AM To: Albert Braden <albertb(a)synopsys.com>; openstack-discuss(a)lists.openstack.org Subject: Re: Rebooted VM disappears when HV lacks memory On Fri, 2019-12-06 at 02:22 +0000, Albert Braden wrote: > Update: It looks like attempting to migrate any VM, alive or dead, from an oversubscribed hypervisor throws the same > error. Maybe I should have done the migrations before setting cpu_allocation_ratio and ram_allocation_ratio. > > Is there a way to recover from this besides deleting VMs from the oversubscribed hypervisors? you could tempoerally set the allocation ratio higher to allow the vm to boot then try migrating it again. you could also maybe try shelving the vm, waiting for ti to be shelve offloaded then unshelve it. > > -----Original Message----- > From: Albert Braden > Sent: Thursday, December 5, 2019 6:08 PM > To: openstack-discuss(a)lists.openstack.org > Subject: RE: Rebooted VM disappears when HV lacks memory > > We changed cpu_allocation_ratio and ram_allocation_ratio to 1 on the hypervisors. Now I'm trying to clean up the mess. > > I'm trying to cold-migrate dead VMs from an oversubscribed hypervisor, but they are failing to migrate: > > root@us01odc-p02-ctrl1:~# os hypervisor list --long|grep hv199 > > 218 | us01-p02-hv199.internal.synopsys.com | QEMU | 10.195.54.226 | up | 138 | 80 > > | 1122304 | 772691 | > > root@us01odc-p02-ctrl1:~# openstack server migrate fc250e76-4ca5-44b1-a6a0-8a7c7ad48c24 > No valid host was found. No valid host found for cold migrate (HTTP 400) (Request-ID: req-cc048221-abaf-4be0-bc90- > f28eecec3df5) > > But we have lots of hosts with space, and if I build a new VM with the same flavor it works. When I check the logs, it > appears that the migration fails because the hypervisor we are moving from is oversubscribed: > > /var/log/nova/nova-conductor.log:2019-12-05 17:30:17.163 48989 WARNING nova.scheduler.client.report [req-cc048221- > abaf-4be0-bc90-f28eecec3df5 5b288691527245bda715ab7744a193e9 deea2d8541f741eda6fb0d242d16bb23 - default > 824941b2ad6a43d7984fab9390290f18] Unable to post allocations for instance 87608240-1ad2-45fc-b4f0-c118e3bf0262 (409 > {"errors": [{"status": 409, "request_id": "req-e56d29f8-c97e-41ac-8481-0598f3c681b2", "detail": "There was a conflict > when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on > resource provider 'fde477cd-fd3d-4296-838c-f9c54b3e97ef'. The requested amount would exceed the capacity. ", "title": > "Conflict"}]}) > > 'fde477cd-fd3d-4296-838c-f9c54b3e97ef' is the UUID of hv199: > > root@us01odc-p02-ctrl1:~# curl -s -X GET -H "X-Auth-Token: $token" -H "Content-Type: application/json" " > http://us01odc-p02-lb.internal.synopsys.com:28778/resource_providers/fde477…";echo > {"generation": 40, "uuid": "fde477cd-fd3d-4296-838c-f9c54b3e97ef", "links": [{"href": "/resource_providers/fde477cd- > fd3d-4296-838c-f9c54b3e97ef", "rel": "self"}, {"href": "/resource_providers/fde477cd-fd3d-4296-838c- > f9c54b3e97ef/inventories", "rel": "inventories"}, {"href": "/resource_providers/fde477cd-fd3d-4296-838c- > f9c54b3e97ef/usages", "rel": "usages"}], "name": "us01-p02-hv199.internal.synopsys.com"} > > What am I doing wrong? Do I need to move the live VMs away from hv199 to make room for the dead ones? > > > > -----Original Message----- > From: Sean Mooney <smooney(a)redhat.com> > Sent: Wednesday, December 4, 2019 4:30 AM > To: Albert Braden <albertb(a)synopsys.com>; openstack-discuss(a)lists.openstack.org > Subject: Re: Rebooted VM disappears when HV lacks memory > > On Wed, 2019-12-04 at 00:51 +0000, Albert Braden wrote: > > We are running Rocky and we have the allocation bug because we set cpu_allocation_ratio and ram_allocation_ratio to > > 1 > > on the controllers but not on the hypervisors, so the hypervisors are oversubscribed. When we try to reboot a VM > > when > > the HV is short memory, the VM is deleted from the HV. It still exists in Openstack but not on the HV. Is it > > possible > > to recover a VM from this state? > > so the vm is likely not deleted the harddisk and the libvirt domain will still be defiend. my guess is starting the > vm cause the oom reaper to kill the qemu process when it was relaunched. > > if you are deploying with memory over subsription enabled you should configre enough swap space equal to total > memory* ram_allocation_ratio. > > if you do that you you should be able to start the vm. now the other option is while the vm is stopped you can > cold migrate it to another host and then start it there.

1 0

openstack volume snapshots and deleting
by Ratliff, John 06 Dec '19

06 Dec '19

If I create a volume, snapshot it, and create a new volume from that snapshot, I cannot seem to delete the snapshot until the volume is deleted. Is this intentional, or am I doing something wrong? Is the new volume based on the snapshot, and that's why I can't delete it? We're using CEPH as a storage backend if that makes a difference. openstack volume snapshot create -volume base-volume base-ss openstack volume create -snapshot base-ss new-volume openstack volume snapshot delete base-ss The last command is the one that fails. It gives no error message. It just doesn't actually delete the snapshot. But If I do this: openstack volume delete new-volume openstack volume snapshot delete base-ss Then it works. I also tried creating a snapshot, creating a volume from the snapshot, then creating a volume from that volume, but I still cannot delete the snapshot until both volumes are deleted. I do not want everlasting snapshots, so I hope I'm doing something wrong that someone can point out. Thanks. --John

3 6

openstack cli server stop command
by Ratliff, John 06 Dec '19

06 Dec '19

Does the openstack server stop command do an ACPI shutdown or does it do the equivalent of pulling the power plug? I want to know if it's safe to run this command rather than logging into the machine and running shutdown (for say a RHEL7 instance). If it does an ACPI shutdown, what happens if the machine doesn't respond? Is there a command to force the machine off? Thanks. --John

2 2

[nova][cinder][ops] question/confirmation of legacy vol attachment migration
by Matt Riedemann 06 Dec '19

06 Dec '19

Hello Cinderinos, I've now got a working patch that migrates legacy volume attachments to new style v3 attachments [1]. The fun stuff showing it working is in this paste [2]. We want to do this data migration in nova because we still have a lot of compatibility code since Queens for pre-v3 style attachments and we can't remove that compatibility code (ever) if we don't first make sure we provide a data migration routine for operators to roll through. So for example if this lands in Ussuri we can can enforce a nova-status upgrade check in V and rip out code in X. Without digging into the patch, this is the flow: 1. On nova-compute restart, query the nova DB for instances on the compute host with legacy volume attachments. 2. For each of those, create a new style attachment with the host connector and update the BlockDeviceMapping information in the nova DB (attachment_id and connection_info). 3. Delete the existing legacy attachment so when the server is deleted the volume status goes back to 'available' due to proper attachment reference counting in the Cinder DB. My main question is on #3. Right now I'm calling the v3 attachment delete API rather than the v2 os-terminate_connection API. Is that sufficient to cleanup the legacy attachment on the storage backend even though the connection was created via os-initialize_connection originally? Looking at the cinder code, attachment_delete hits the connection terminate code under the covers [3]. So that looks OK. The only thing I can really think of is if a host connector is not provided or tracked with the legacy attachment, is that going to cause problems? Note that I think volume drivers are already required to deal with that today anyway because of the "local delete" scenario in the compute API where the compute host that the server is on is down and thus we don't have a host connector to provide to Cinder to terminate the connection. So Cinder people, are you OK with this flow? Hello Novaheads, Do you have any issues with the above? Note the migration routine is threaded out on compute start so it doesn't block, similar to the ironic flavor data migration introduced in Pike. One question I have is if we should add a config option for this so operators can enable/disable it as needed. Note that this requires nova to be configured with a service user that has the admin role to do this stuff in cinder since we don't have a user token, similar to nova doing things with neutron ports without a user token. Testing this with devstack requires [4]. By default [cinder]/auth_type is None and not required so by default this migration routine is not going to run so maybe that is sufficient? Hello Operatorites, Do you have any issues with what's proposed above? [1] https://review.opendev.org/#/c/549130/ [2] http://paste.openstack.org/show/781063/ [3] https://github.com/openstack/cinder/blob/410791580ef60ddb03104bf20766859ed9… [4] https://review.opendev.org/#/c/685488/ -- Thanks, Matt

5 10

email alert settings are not applied (ask.openstack.org)
by Eugen Block 06 Dec '19

06 Dec '19

Hi all, I'm not really sure if this is the right place to ask but I'll try it anyway. I'm regularly checking ask.openstack.org for interesting questions and sometimes I comment and respond. In my profile I configured instant email notification if someone comments or responds to my questions/comments. But this has never worked properly, I think this worked just once two or three years ago, never again. The only setting that seems to work is the weekly notification about new questions. Can anyone comment on this? Should I try to change the email address and see if that works? Any insight is appreciated! Regards, Eugen

2 2

[horizon] Shanghai PTG recap
by Akihiro Motoki 06 Dec '19

06 Dec '19

Hi, I am summarizing the horizon PTG in Shanghai. One month passed from the PTG, but I believe the summary is still useful :-) Eight folks joined the session and we had productive discussions. Thanks all! - PTG etherpad https://etherpad.openstack.org/p/horizon-u-ptg - release priority etherpad https://etherpad.openstack.org/p/horizon-release-priorities [Priorities for Ussuri] After the PTG, we discussed the project priorities for Ussuri based on the PTG discussion and agreed the following as project priorities. We will cover them in the weekly team meetings. * High * Catch up the new policy mechanism * Bump and clean up Django versions * Error message refactoring * Medium * xstatic updates * ini-based-configuration * Chinese translation handling [PTG recap] * Train retrospective * To keep * We succeeded to restore integration tests in horizon and several plugins. They just cover basic GET scenarios. Most JS tests do not cover scenario testing so it would be a good step to avoid plugin breakage. * We can keep the project healthiness to some level regardless of the low number of contributors. * To improve * The bundle policy files in horizon is out of date from back-end services and there was no progress since Stein release. For example, the new policies (system scope, deprecated policies) from the keystone repo does not work with horizon. It should be priorities in Ussuri release. We will join the pop-up team for the policy enhancement. * Some community goal from queens has not completed. (Most of them have done.) * Supported versions of Django and Python * We discussed the plan to enable Django 2.2 (the current LTS) by default and drop Django 1.11 support. The current plan is to bump the default Django to 2.2 around milestone-1 and drop Django 1.11 support by milestone-2. The minimum goal is to support Django 2.2 as it will be the only LTS supported by Django community as of Ussuri release. * The version changes in Django may break horizon plugins so we need enough time to test them in real deployments. * Integration test coverage would help the transition of Django version and let's try integration tests more with our best. * X-Static updates * Many xstatic packages are not updated for long. They has has fixes including security fixes, so we need to update them. radomir volunteered for the effort. * JS loading * e0ne shared his experiences on his trial to implement a plugin using react. * The problem is that it is not easy to use newer JS libraries. An example of problmes with a current approach is https://review.opendev.org/#/c/652657/ * It comes from the gap between the xstatic way to use same versions for all JS applications and modern JS packaging like webpack. * There is no good conclusion in this topic, but a long-term goal would be to use modern packaging mechanism instead of xstatic. It could be an issue for distribution packagers. * Horizon plugins maintenance * This slot covered topics around horizon plugin ecosystem and their maintenance. * The main outcomes are: * we agreed to change the release model of horizon to cycle-with-intermediary. Horizon plugins use horizon as library and we need to release horizon promptly whenever we have changes which affect plugins. cycle-with-intermediary model would fit more for this purpose. * A concern on that multiple major version bump potentially happens was raised, but we agreed that the version number does not matter and the best practice to avoid this is to drop deprecated features at the beginning of the cycle. * horizon team asked individual project teams with horizon plugins to give +2 right to horizon cores to make plugin maintenance smooth. * Most team agreed and we had conversations with more team at the PTG. * One important note is that horizon team should shouldn’t merge anything to plugins without +2 from plugin team. * We will update our documentation to cover this change in the plugin maintenance and e0ne volunteered for it. * Some side-discussion happened. There was no decision but they might be worth considering in future. * An idea to split nova/cinder/neutron/keystone/.... support to plugins and make them mandatory dependencies. * Use of micro-frontend concept * Chinese translation handling * From horizon perspective we agreed that we would like to move to the new locale names (zh-hans/hant) both in horizon (+plugins). * The main questions are the transition plan and locales for documentations. Further discussion with the i18n team happened in Shanghai. The summary of the discussion will be coming soon, but the first step agreed is that the horizon team confirms the new locales work fine in horizon. * Feature Gaps * Brief discussions happened for features with volunteers. * We will visit the status of the gaps in the etherpad https://etherpad.openstack.org/p/horizon-feature-gap * Refactor Error Messages * We received a good number of bug reports which complain that horizon error messages are too general and hide error details. * We agree to show both error message summary (translated) and detail message from API. * An idea discussed is to have a collapsible box to show an error detail along with a summary message. This approach would bring us a good balance of translated message and error detail from back-end services. * It would be a nice improvement in Ussuri. * Deprecations and Removals * Short discussion covered potential feature deprecation and removals. * Cinder v2 support will be dropped soon. * ini-based-configuration * amotoki expressed that he continues the effort and hopefully completes it in Ussuri. * API versions of backend services * horizon uses the minimum microversions to communicate with back-end services unless new versions are required to support features. * The possibility to bump the minimum versions used (per cycle?) was discussed and we agreed to keep the current strategy considering dev resources and maintenance. * Adoption of openstacksdk was discussed. openstacksdk supports older APIs and it would help us keep the current strategy on API versions used. * To use openstacksdk, we need to explore how to instantiate keystonauth Auth object with a token (as horizon stores only token and does not store auth information). We can start the investigation on this and e0ne volunteered. That's all we discussed in Shanghai. I hope we will have good progress in Ussuri. Best Regards, Akihiro Motoki (irc: amotoki)

1 0

[ptl] Train cycle-trailing release deadline
by Sean McGinnis 06 Dec '19

06 Dec '19

Hello all, For those teams with cycle-trailing deliverables, this is a reminder to make sure to get your Train series deliverables released. We like to have all of those out within three months of the coordinated release. For reference, here is a list of deliverables we have marked as Train cycle-trailing in the releases repo that do not appear to have a final release: cinderlib kayobe kolla-ansible kolla-cli kolla openstack-ansible tripleo-ui If these, or any others, need to be added or changed for Ussuri, please propose a placeholder deliverable file under deliverables/ussuri or update any existing ones with any needed changes. Thanks! Sean

3 3

Rebooted VM disappears when HV lacks memory
by Albert Braden 06 Dec '19

06 Dec '19

2 4

[magnum][stein] broken pipe
by Ignazio Cassano 05 Dec '19

05 Dec '19

Hello, I installed magnum stein on centos 7 but magnum-api reports the following error in /var/log/messages: Dec 5 21:14:49 tst2-osctrl01 magnum-api: Traceback (most recent call last): Dec 5 21:14:49 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/SocketServer.py", line 568, in process_request Dec 5 21:14:49 tst2-osctrl01 magnum-api: self.finish_request(request, client_address) Dec 5 21:14:49 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request Dec 5 21:14:49 tst2-osctrl01 magnum-api: self.RequestHandlerClass(request, client_address, self) Dec 5 21:14:49 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/SocketServer.py", line 651, in __init__ Dec 5 21:14:49 tst2-osctrl01 magnum-api: self.finish() Dec 5 21:14:49 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/SocketServer.py", line 710, in finish Dec 5 21:14:49 tst2-osctrl01 magnum-api: self.wfile.close() Dec 5 21:14:49 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/socket.py", line 279, in close Dec 5 21:14:49 tst2-osctrl01 magnum-api: self.flush() Dec 5 21:14:49 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/socket.py", line 303, in flush Dec 5 21:14:49 tst2-osctrl01 magnum-api: self._sock.sendall(view[write_offset:write_offset+buffer_size]) Dec 5 21:14:49 tst2-osctrl01 magnum-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 401, in sendall Dec 5 21:14:49 tst2-osctrl01 magnum-api: tail = self.send(data, flags) Dec 5 21:14:49 tst2-osctrl01 magnum-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 395, in send Dec 5 21:14:49 tst2-osctrl01 magnum-api: return self._send_loop(self.fd.send, data, flags) Dec 5 21:14:49 tst2-osctrl01 magnum-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 382, in _send_loop Dec 5 21:14:49 tst2-osctrl01 magnum-api: return send_method(data, *args) Dec 5 21:14:49 tst2-osctrl01 magnum-api: error: [Errno 32] Broken pipe Anyone could help me please ? Regards Ignazio

1 0

[goals][Drop Python 2.7 Support] Week R-23 Update
by Ghanshyam Mann 05 Dec '19

05 Dec '19

Hello Everyone, Below is the progress on "Drop Python 2.7 Support" for R-23 week. We are in phase-1 which targets all the OpenStack services to drop the py2 support. Schedule: https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html#sc… NOTE: Phase-2 is for Library (including client library), QA tooling. Phase-3 is mainly for openstack/requirement and audit to all repo. (we can always adjust the schedule to make it smooth migration. For example, dropping a few testing jobs from phase-2/3 candidates if any cross-testing jobs are failing due to phase-1 candidates dropping the support[1].) Summary: ======= Patches to all services repo have been up for review[2]. The OpenStack services have not merged the py2 drop patches: * Adjutant * Barbican * Designate * ec2-api * Glance * Neutron * Freezer * Kolla * Monasca * Karbor * Congress * Telemetry * Storlets * Sahara * Blazar * Masakari * Placement * Heat (heat-dashboard patch is note merged ) * Tacker * Trove * Zaqar * Quinling * Manila * Tricircle * openstack-helm * python-openstackclient The OpenStack services dropped the py2 support: * Cinder * Nova * Ironic * Keystone * Octavia * Watcher * Vitrage * Zun * Winstackers * Mistral * Cyborg * Kuryr * Cloudkitty * Murano * Solum * Magnum * Senlin * Searchlight * openstack-ansible - I do not think we need any update in openstack-ansible? release notes jobs etc are already on py3. Rest all projects are targetted for phase-2 (after m-1). How you can help: ============== - Review the patches. Push the patches if I missed any repo targetted for phase-1. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011392… [2] https://review.opendev.org/#/q/topic:drop-py27-support -gmann

1 0

[release] Release countdown for week R-22
by Sean McGinnis 05 Dec '19

05 Dec '19

Development Focus ----------------- The Ussuri-1 milestone is next week, on December 12! Project team plans for the Ussuri cycle should now be solidified. General Information ------------------- If you planned to change the release model for any of your deliverables this cycle, please remember to do so ASAP, before milestone-1. Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library which had changes but has not been otherwise released since the Train release. PTL's or release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well, by posting a -1. If we do not hear anything at all by the end of the week, we will assume things are OK to proceed. NB: If one of your libraries is still releasing 0.x versions, start thinking about when it will be appropriate to do a 1.0 version. The version number does signal the state, real or perceived, of the library, so we strongly encourage going to a full major version once things are in a good and usable state. Upcoming Deadlines & Dates -------------------------- Ussuri-1 milestone: December 12 (R-22 week)

1 0

[all][requirements][drop-py27-support] Requirements is dropping cross tests with projects that will be dropping support for python2.7 this cycle
by Matthew Thode 05 Dec '19

05 Dec '19

Hi all, The requirements project is dropping python2.7 cross tests we do for those projects that are also dropping python2.7 support this cycle. We are doing this as wel do not know when projects will drop support (neutron seems to have done so, cinder is soon as well). This was originally scheduled for later but need to be pushed up as we missed the dependency requirements has on those projects for cross jobs. Here's the review if you are curious :D https://review.opendev.org/697377 -- Matthew Thode

3 2

How Bank or Financial sector benefited from OpenStack ?
by Mobarak Hossain 05 Dec '19

05 Dec '19

Hi, I want to approach to Bank for Open Infrastructure . They are asking how a Bank get benefited from Open Infrastructure .Except open Source . Is there any special features for Bank . I saw a video that Amex is using . Is there suggestion for me please let me know . Thanks -- Mobarak Hossain

2 1

[OSSN-0085] Cinder configuration option can leak secret key from Ceph backend
by Brian Rosmaita 05 Dec '19

05 Dec '19

Cinder configuration option can leak secret key from Ceph backend --- ### Summary ### This vulnerability is present only when Cinder has a Ceph backend *and* the ``rbd_keyring_conf`` option in the Cinder configuration file is set. (The option is unset by default.) Under these circumstances, the keyring content may be leaked to a regular user. ### Affected Services / Software ### Cinder, Pike, Queens, Rocky, Stein, Train ### Discussion ### When the ``rbd_keyring_conf`` option is set, a user who creates a volume and then does a local attach of that volume using the Cinder REST API, may discover the keyring content for the Ceph cluster. (This does not apply to the normal Nova attach process. The user must contact the Cinder REST API directly for this exploit.) If this user then gains access to the Ceph cluster independently of Cinder, these credentials may be used to access any volume in the cluster. This issue was reported by Raphael Glon of OVH. NOTE: This issue is different from OSSA-2019-003, through which Ceph credentials could be leaked from the Nova REST API during error conditions, but we suggest you take this opportunity to review that security advisory and make sure your system has been updated or patched to address it: https://security.openstack.org/ossa/OSSA-2019-003.html ### Recommended Actions ### Any installation currently using the ``rbd_keyring_conf`` option should immediately stop using that option. In order for Cinder backups to continue working, the Ceph keyring secrets should be deployed directly on cinder-backup hosts in their standard location: /etc/ceph/<cluster_name>.client.<user_name>.keyring The ``rbd_keyring_conf`` is deprecated in the Ussuri release and will be removed early in the 'V' development cycle. There is no replacement planned for this functionality. ### Contacts / References ### Author: Brian Rosmaita, Red Hat This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0085 Original LaunchPad Bug : https://bugs.launchpad.net/cinder/+bug/1849624 Mailing List : [Security] tag on openstack-discuss(a)lists.openstack.org OpenStack Security Project : https://launchpad.net/~openstack-ossg

1 0

What's the best recommendation of cache solution for Openstack services?
by 양유석 05 Dec '19

05 Dec '19

Hi, stackers. I wonder how other system deal with the problem we encountered. Any advice would be appreciated. He had a over 1000 hypervisors and hit some connection bottleneck in a DB. The DB instance handle all Openstack services (nova, neutron, cinder, glance ...) Also, we have some additional systems (kubernetes, consul...) which periodically request openstack API. Since most of request from additional systems is READ, I am trying to find cache approach for Openstack service. Although some services like keystone has their own caching layer, I could not find general solutions. Is there any project to solve this problem (reduce DB overhead in a lot of read overhead) I found oslo.cache only provide decorator for a function, so wonder there is any service level approach. Thanks

2 1

DHCP timeout when creating instances for specific tenants
by Grant Morley 05 Dec '19

05 Dec '19

Hi all, I wonder if anyone can help shed any light on an odd issue we are seeing with only a couple of specific tenants. Basically if they launch an instance they are taking about 5 minutes to launch rather than our usual 30 second or so launch. We are seeing the following on the instance logs: https://pastebin.com/hDstsd8G Weirdly it only seems to be happening for 1 or 2 new tenants. I have tested this on our personal account and a few other customers have tested and their instances launch really quickly as expected. Is there anything specific during the tenant creation that can cause this issue? Or are there any logs in nova / neutron I should be looking out for that might shed some light? I haven't seen anything that is obvious. Any help would be much appreciated as we are a little stumped at the moment. Many thanks, -- Grant Morley Cloud Lead, Civo Ltd www.civo.com <https://www.civo.com/>| Signup for an account! <https://www.civo.com/signup>

4 8

[neutron] Proposing Jakub Libosvar as Neutron core reviewer
by Slawek Kaplonski 05 Dec '19

05 Dec '19

Hi neutrinos, We already started process of migrating networking-ovn driver to be one of in-tree neutron drivers. Blueprint for that is [1]. As part of this process I today proposed to include networking-ovn-core group into neutron-core group. Mail about it can be found at [2]. One of persons in networking-ovn-group is Jakub Libosvar who was Neutron core for very long time in the past. He knows very well not only ovn related code but also have great knowledge about all Neutron code base. So I would like to propose to Jakub as Neutron core reviewer again as he will be back working on neutron again now, after ovn will be in-tree driver. What do You think about it? I will wait for Your opinions for 1 week from now. Thx for all Your comments about it. [1] https://blueprints.launchpad.net/neutron/+spec/neutron-ovn-merge [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011240… — Slawek Kaplonski Senior software engineer Red Hat

7 7

[heat][heat-translator] add Hiroyuki Jo to heat-translator core reviewer
by Rico Lin 05 Dec '19

05 Dec '19

Hi all Hiroyuki Jo has been active in the past few months. Also, he raises his willingness to help maintain patches in the last PTG [1]. It's my pleasure to add Hiroyuki Jo to heat-translator-core team with condition that me and other core members should help to review his patch (so no self-approval) and provide review guideline if needed. Free feel to suggest otherwise. [1] https://etherpad.openstack.org/p/PVG-heat -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

1 0

Swedish University Engagement Program
by Kendall Nelson 05 Dec '19

05 Dec '19

Hello Everyone! During the PTG, I talked to Kim Hindhart (along with Ildiko Vancsa, Sean McGinnis, and Victoria Martinez de la Cruz) about a program he was working on with several universities in Sweden (Blekinge Institute of Technology, Uppsala University, and KTH Royal Institute of Technology) to get students involved in open source, specifically involved in OpenStack. While this program is still getting off the ground, with its first round to start in this coming February with students working on Horizon (led by Kim), we are hoping to collect work for the second round that will be more student driven. The only disclaimer is that items suggested might not get picked up until the fall semester in September. I am collecting low-hanging-fruit here on this etherpad[1].Hopefully you have a lot of things tagged as such on Launchpad/StoryBoard already and can easily add those links to the etherpad. Can't wait to see what we can come up with! -Kendall (diablo_rojo) [1] https://etherpad.openstack.org/p/swedish-university-program-work

1 0

[tripleo][ppc64le] Unexpected Return Code from TripleO Container Build Playbooks on ppc64le
by Adam Kimball 04 Dec '19

04 Dec '19

Hello all, Mike Turek and I have been working to get a ppc64le job to build tripleo containers for quite some time now. A few weeks back, we stumbled into a strange issue where the containers build job was issuing a return code 1, despite all containers successfully building. This has largely blocked our ability to push containers up to docker hub or report success through the delorean API. We've been pushing a number of patches to expand the logging of both our job, and the tripleo playbooks themselves, to get a better idea of what's going on. The most recent was a patch to show which containers have python futures that have been throwing exceptions, and why. [1] The end result of this seems to be that a number of jobs are reporting as incomplete. An example of this can be seen at timestamp 20:16:07 of an example build log. [2] However, upon checking the list of successfully built containers [3], or the RDO registry itself [4], one can see that the containers producing job not complete errors have actually built, and are being uploaded. The error log generated by the tripleo playbooks is also empty. [5] At this point, we're wondering what the path forward is. It seems like the issue stems from some unintended behavior in the tripleo playbooks themselves, not anything from our job. We're trying to figure out if this behavior is something that should be preventing us from reporting successful builds, and if so, how it can be fixed. Thanks, Adam Kimball [1] - https://review.opendev.org/#/c/695723/ [2] - https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master… [3] - https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master… [4] - https://console.registry.rdoproject.org/registry [5] - https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master…

2 1

[cinder] Ussuri Shanghai and Virtual PTG update
by Brian Rosmaita 04 Dec '19

04 Dec '19

The PTG summary document has been updated with a quick synopsis of what we discussed at the Virtual PTG last week: https://wiki.openstack.org/wiki/CinderUssuriPTGSummary All sessions (both Shanghai and Virtual) were recorded. The above page contains links to the recordings, which you can also find on the Cinder YouTube channel. Please look through the summary document to remind yourself about any actions you volunteered for. Additionally, there's an etherpad of unclaimed action items if you're looking for something to do: https://etherpad.openstack.org/p/cinder-ussuri-ptg-actions Thanks to everyone who participated in the PTG events. cheers, brian

1 0