December 2019 - openstack-discuss

【ironic】 booting from volume
by 海风吹 09 Dec '19

09 Dec '19

hi:    The pike version of ironic support the feature: booting from volume. There is a question: when booting from volume, the baremetal node should be reachable to the storage node,but when deploy done, the baremetal's network  completely belong to the vpc. How could the baremetal node booting from volume??     I should add the extar storage nic to communicate with storage node??        expecting for the answer thanks guo

1 0

High CPU utilization by systemd process || debian 8
by Md. Farhad Hasan Khan 09 Dec '19

09 Dec '19

Hi, We are observing high CPU utilization by systemd process. 100% CPU utilize by systemd process when no user is being logged in to the server but CPU utilization became 0% while any user logged in to the server. What cloud be possible reason behind this issue and solution? Log collected via login to the server via ssh or console: (0% CPU utilization) top -b -n 1 | grep systemd | grep 612 612 root 20 0 3386692 2.051g 4932 T 0.0 6.5 22029:26 systemd Log collected without login to the server : (100% CPU utilization) # ssh user(a)192.168.18.11 <mailto:user@192.168.18.11> top -b -n 1 | grep systemd | grep 612 user(a)192.168.18.11's <mailto:user@192.168.18.11's> password: 612 root 20 0 3386692 2.051g 4932 S 789.2 6.5 22031:31 systemd Please find the server information: lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 8.11 (jessie) Release: 8.11 Codename: jessie uname -r 3.16.0-6-amd64 Thanks & B’Rgds, Rony

2 1

[magnum] [stein] kubernetes stack does not terminate
by Ignazio Cassano 08 Dec '19

08 Dec '19

hello, I've just installed stein with magnum on centos 7. I am creating a kubernetes cluster but it is running since 22 minutes and I think it will not work because the heat stack is waiting OSSoftwareDeployment completes I presume something is not working on heat with magnum. I ran several stack with Software Deployment without magnum, and they worked fine Thanks Ignazio

2 16

RE: High CPU utilization by systemd process || debian 8
by Md. Farhad Hasan Khan 07 Dec '19

07 Dec '19

Hi, Please help to solve this. Thanks/Rony From: Md. Farhad Hasan Khan [mailto:rony.khan@brilliant.com.bd] Sent: Saturday, December 7, 2019 11:55 AM To: 'OpenStack Discuss' Subject: High CPU utilization by systemd process || debian 8 Hi, We are observing high CPU utilization by systemd process. 100% CPU utilize by systemd process when no user is being logged in to the server but CPU utilization became 0% while any user logged in to the server. What cloud be possible reason behind this issue and solution? Log collected via login to the server via ssh or console: (0% CPU utilization) top -b -n 1 | grep systemd | grep 612 612 root 20 0 3386692 2.051g 4932 T 0.0 6.5 22029:26 systemd Log collected without login to the server : (100% CPU utilization) # ssh user(a)192.168.18.11 <mailto:user@192.168.18.11> top -b -n 1 | grep systemd | grep 612 user(a)192.168.18.11's <mailto:user@192.168.18.11's> password: 612 root 20 0 3386692 2.051g 4932 S 789.2 6.5 22031:31 systemd Please find the server information: lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 8.11 (jessie) Release: 8.11 Codename: jessie uname -r 3.16.0-6-amd64 Thanks & B’Rgds, Rony

1 0

neutron-metadata-agent broken pipe
by Albert Braden 07 Dec '19

07 Dec '19

As our production cluster grows in size we are starting to have trouble with neutron-metadata-agent. After restarting it is happy for a minute and then it complains "2019-12-06 17:54:24.615 664587 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 11, greater than warning threshold: 10. There could be a leak. Increasing threshold to: 20" It increases the threshold a couple of times and then after increasing to 40 we start to see errors: 2019-12-06 17:55:10.119 664578 INFO eventlet.wsgi.server [-] Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 521, in handle_one_response write(b''.join(towrite)) File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 462, in write wfile.flush() File "/usr/lib/python2.7/socket.py", line 307, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 390, in sendall tail = self.send(data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 384, in send return self._send_loop(self.fd.send, data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 371, in _send_loop return send_method(data, *args) error: [Errno 32] Broken pipe It looks like increasing the threshold to 40 fails because it keeps trying: 2019-12-06 17:55:17.452 664597 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 21, greater than warning threshold: 20. There could be a leak. Increasing threshold to: 40 And the errors increase until the log is nothing but errors, and VMs fail to boot. root@us01odc-p02-ctrl1:~# tail -f /var/log/neutron/neutron-metadata-agent.log self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 390, in sendall tail = self.send(data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 384, in send return self._send_loop(self.fd.send, data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 371, in _send_loop return send_method(data, *args) error: [Errno 32] Broken pipe 2019-12-06 17:59:23.664 664597 INFO eventlet.wsgi.server [-] 10.195.73.174,<local> "GET /latest/meta-data/instance-id HTTP/1.0" status: 200 len: 0 time: 62.1517761 2019-12-06 17:59:23.756 664583 INFO eventlet.wsgi.server [-] Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 521, in handle_one_response write(b''.join(towrite)) File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 462, in write wfile.flush() File "/usr/lib/python2.7/socket.py", line 307, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 390, in sendall tail = self.send(data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 384, in send return self._send_loop(self.fd.send, data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 371, in _send_loop return send_method(data, *args) error: [Errno 32] Broken pipe 2019-12-06 17:59:23.757 664583 INFO eventlet.wsgi.server [-] 10.195.65.69,<local> "GET /latest/meta-data/instance-id HTTP/1.0" status: 200 len: 0 time: 63.0419171 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent [-] Unexpected error.: MessagingTimeout: Timed out waiting for a reply to message ID 77551d4cf4394b7b9cdfad68c0be46e8 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent Traceback (most recent call last): 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 89, in __call__ 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent instance_id, tenant_id = self._get_instance_and_tenant_id(req) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 162, in _get_instance_and_tenant_id 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent ports = self._get_ports(remote_address, network_id, router_id) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 155, in _get_ports 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent return self._get_ports_for_remote_address(remote_address, networks) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/common/cache_utils.py", line 116, in __call__ 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent return self.func(target_self, *args, **kwargs) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 137, in _get_ports_for_remote_address 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent ip_address=remote_address) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 106, in _get_ports_from_server 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent return self.plugin_rpc.get_ports(self.context, filters) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 72, in get_ports 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent return cctxt.call(context, 'get_ports', filters=filters) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 173, in call 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent time.sleep(wait) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent self.force_reraise() 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent six.reraise(self.type_, self.value, self.tb) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 150, in call 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent return self._original_context.call(ctxt, method, **kwargs) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 179, in call 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent retry=self.retry) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 133, in _send 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent retry=retry) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent call_monitor_timeout, retry=retry) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent call_monitor_timeout) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent message = self.waiters.get(msg_id, timeout=timeout) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent 'to message ID %s' % msg_id) 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent MessagingTimeout: Timed out waiting for a reply to message ID 77551d4cf4394b7b9cdfad68c0be46e8 2019-12-06 17:59:23.776 664595 ERROR neutron.agent.metadata.agent 2019-12-06 17:59:23.778 664595 INFO eventlet.wsgi.server [-] Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 521, in handle_one_response write(b''.join(towrite)) File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 462, in write wfile.flush() File "/usr/lib/python2.7/socket.py", line 307, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 390, in sendall tail = self.send(data, flags) File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 384, in send return self._send_loop(self.fd.send, data, flags) What's causing this? Are we overloading RMQ?

1 0

[nova][ironic] Changing an owner of a provisioned node
by Matt Riedemann 07 Dec '19

07 Dec '19

The 1.50 microversion [1] in the ironic API added the "owner" field to the node and I'm trying to use that to add some scheduler filtering in nova [2]. It's my understanding that the owner field on a provisioned node (instance_uuid on the node is set) can be changed, but I'm surprised that is allowed. Was that an oversight in developing that feature? The use case for the scheduler filter is baremetal nodes are owned by different (non-admin) projects in a deployment. When a non-admin project creates a baremetal server via nova, nova will filter out nodes that are not owned by the project (based on the node.owner field). If a node isn't owned by any project, only admins can use it. Admins also have access to all nodes regardless of owner. Given that, let's say user 1 from project A creates a server on nova X that is owned by project A (node.owner=A). Then the node.owner is changed to project B. What should happen? Should nova detect that ownership change and stop the node or something? Note that with other resources that can transfer ownership, like volumes, that can only be done when they aren't in use. So why don't we have the same rules for nodes? Assuming we do want to enforce this in the API (a 409 response when trying to change the owner on a provisioned node), how would that be done given this is a problem since 1.50 which was added in Stein? Would a policy rule be added to ironic to determine if someone can change the owner on a provisioned node and if so, what would be the default rule? The same as "baremetal:node:update" (rule:is_admin)? [1] https://docs.openstack.org/ironic/latest/contributor/webapi-version-history… [2] https://blueprints.launchpad.net/nova/+spec/ironic-tenant-filter -- Thanks, Matt

3 3

RE: Rebooted VM disappears when HV lacks memory
by Albert Braden 06 Dec '19

06 Dec '19

Hi Sean, I didn't want to mess with prod configs but shelve/unshelve works. Thanks for your advice! -----Original Message----- From: Sean Mooney <smooney(a)redhat.com> Sent: Friday, December 6, 2019 3:53 AM To: Albert Braden <albertb(a)synopsys.com>; openstack-discuss(a)lists.openstack.org Subject: Re: Rebooted VM disappears when HV lacks memory On Fri, 2019-12-06 at 02:22 +0000, Albert Braden wrote: > Update: It looks like attempting to migrate any VM, alive or dead, from an oversubscribed hypervisor throws the same > error. Maybe I should have done the migrations before setting cpu_allocation_ratio and ram_allocation_ratio. > > Is there a way to recover from this besides deleting VMs from the oversubscribed hypervisors? you could tempoerally set the allocation ratio higher to allow the vm to boot then try migrating it again. you could also maybe try shelving the vm, waiting for ti to be shelve offloaded then unshelve it. > > -----Original Message----- > From: Albert Braden > Sent: Thursday, December 5, 2019 6:08 PM > To: openstack-discuss(a)lists.openstack.org > Subject: RE: Rebooted VM disappears when HV lacks memory > > We changed cpu_allocation_ratio and ram_allocation_ratio to 1 on the hypervisors. Now I'm trying to clean up the mess. > > I'm trying to cold-migrate dead VMs from an oversubscribed hypervisor, but they are failing to migrate: > > root@us01odc-p02-ctrl1:~# os hypervisor list --long|grep hv199 > > 218 | us01-p02-hv199.internal.synopsys.com | QEMU | 10.195.54.226 | up | 138 | 80 > > | 1122304 | 772691 | > > root@us01odc-p02-ctrl1:~# openstack server migrate fc250e76-4ca5-44b1-a6a0-8a7c7ad48c24 > No valid host was found. No valid host found for cold migrate (HTTP 400) (Request-ID: req-cc048221-abaf-4be0-bc90- > f28eecec3df5) > > But we have lots of hosts with space, and if I build a new VM with the same flavor it works. When I check the logs, it > appears that the migration fails because the hypervisor we are moving from is oversubscribed: > > /var/log/nova/nova-conductor.log:2019-12-05 17:30:17.163 48989 WARNING nova.scheduler.client.report [req-cc048221- > abaf-4be0-bc90-f28eecec3df5 5b288691527245bda715ab7744a193e9 deea2d8541f741eda6fb0d242d16bb23 - default > 824941b2ad6a43d7984fab9390290f18] Unable to post allocations for instance 87608240-1ad2-45fc-b4f0-c118e3bf0262 (409 > {"errors": [{"status": 409, "request_id": "req-e56d29f8-c97e-41ac-8481-0598f3c681b2", "detail": "There was a conflict > when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on > resource provider 'fde477cd-fd3d-4296-838c-f9c54b3e97ef'. The requested amount would exceed the capacity. ", "title": > "Conflict"}]}) > > 'fde477cd-fd3d-4296-838c-f9c54b3e97ef' is the UUID of hv199: > > root@us01odc-p02-ctrl1:~# curl -s -X GET -H "X-Auth-Token: $token" -H "Content-Type: application/json" " > http://us01odc-p02-lb.internal.synopsys.com:28778/resource_providers/fde477…";echo > {"generation": 40, "uuid": "fde477cd-fd3d-4296-838c-f9c54b3e97ef", "links": [{"href": "/resource_providers/fde477cd- > fd3d-4296-838c-f9c54b3e97ef", "rel": "self"}, {"href": "/resource_providers/fde477cd-fd3d-4296-838c- > f9c54b3e97ef/inventories", "rel": "inventories"}, {"href": "/resource_providers/fde477cd-fd3d-4296-838c- > f9c54b3e97ef/usages", "rel": "usages"}], "name": "us01-p02-hv199.internal.synopsys.com"} > > What am I doing wrong? Do I need to move the live VMs away from hv199 to make room for the dead ones? > > > > -----Original Message----- > From: Sean Mooney <smooney(a)redhat.com> > Sent: Wednesday, December 4, 2019 4:30 AM > To: Albert Braden <albertb(a)synopsys.com>; openstack-discuss(a)lists.openstack.org > Subject: Re: Rebooted VM disappears when HV lacks memory > > On Wed, 2019-12-04 at 00:51 +0000, Albert Braden wrote: > > We are running Rocky and we have the allocation bug because we set cpu_allocation_ratio and ram_allocation_ratio to > > 1 > > on the controllers but not on the hypervisors, so the hypervisors are oversubscribed. When we try to reboot a VM > > when > > the HV is short memory, the VM is deleted from the HV. It still exists in Openstack but not on the HV. Is it > > possible > > to recover a VM from this state? > > so the vm is likely not deleted the harddisk and the libvirt domain will still be defiend. my guess is starting the > vm cause the oom reaper to kill the qemu process when it was relaunched. > > if you are deploying with memory over subsription enabled you should configre enough swap space equal to total > memory* ram_allocation_ratio. > > if you do that you you should be able to start the vm. now the other option is while the vm is stopped you can > cold migrate it to another host and then start it there.

1 0

openstack volume snapshots and deleting
by Ratliff, John 06 Dec '19

06 Dec '19

If I create a volume, snapshot it, and create a new volume from that snapshot, I cannot seem to delete the snapshot until the volume is deleted. Is this intentional, or am I doing something wrong? Is the new volume based on the snapshot, and that's why I can't delete it? We're using CEPH as a storage backend if that makes a difference. openstack volume snapshot create -volume base-volume base-ss openstack volume create -snapshot base-ss new-volume openstack volume snapshot delete base-ss The last command is the one that fails. It gives no error message. It just doesn't actually delete the snapshot. But If I do this: openstack volume delete new-volume openstack volume snapshot delete base-ss Then it works. I also tried creating a snapshot, creating a volume from the snapshot, then creating a volume from that volume, but I still cannot delete the snapshot until both volumes are deleted. I do not want everlasting snapshots, so I hope I'm doing something wrong that someone can point out. Thanks. --John

3 6

openstack cli server stop command
by Ratliff, John 06 Dec '19

06 Dec '19

Does the openstack server stop command do an ACPI shutdown or does it do the equivalent of pulling the power plug? I want to know if it's safe to run this command rather than logging into the machine and running shutdown (for say a RHEL7 instance). If it does an ACPI shutdown, what happens if the machine doesn't respond? Is there a command to force the machine off? Thanks. --John

2 2

[nova][cinder][ops] question/confirmation of legacy vol attachment migration
by Matt Riedemann 06 Dec '19

06 Dec '19

Hello Cinderinos, I've now got a working patch that migrates legacy volume attachments to new style v3 attachments [1]. The fun stuff showing it working is in this paste [2]. We want to do this data migration in nova because we still have a lot of compatibility code since Queens for pre-v3 style attachments and we can't remove that compatibility code (ever) if we don't first make sure we provide a data migration routine for operators to roll through. So for example if this lands in Ussuri we can can enforce a nova-status upgrade check in V and rip out code in X. Without digging into the patch, this is the flow: 1. On nova-compute restart, query the nova DB for instances on the compute host with legacy volume attachments. 2. For each of those, create a new style attachment with the host connector and update the BlockDeviceMapping information in the nova DB (attachment_id and connection_info). 3. Delete the existing legacy attachment so when the server is deleted the volume status goes back to 'available' due to proper attachment reference counting in the Cinder DB. My main question is on #3. Right now I'm calling the v3 attachment delete API rather than the v2 os-terminate_connection API. Is that sufficient to cleanup the legacy attachment on the storage backend even though the connection was created via os-initialize_connection originally? Looking at the cinder code, attachment_delete hits the connection terminate code under the covers [3]. So that looks OK. The only thing I can really think of is if a host connector is not provided or tracked with the legacy attachment, is that going to cause problems? Note that I think volume drivers are already required to deal with that today anyway because of the "local delete" scenario in the compute API where the compute host that the server is on is down and thus we don't have a host connector to provide to Cinder to terminate the connection. So Cinder people, are you OK with this flow? Hello Novaheads, Do you have any issues with the above? Note the migration routine is threaded out on compute start so it doesn't block, similar to the ironic flavor data migration introduced in Pike. One question I have is if we should add a config option for this so operators can enable/disable it as needed. Note that this requires nova to be configured with a service user that has the admin role to do this stuff in cinder since we don't have a user token, similar to nova doing things with neutron ports without a user token. Testing this with devstack requires [4]. By default [cinder]/auth_type is None and not required so by default this migration routine is not going to run so maybe that is sufficient? Hello Operatorites, Do you have any issues with what's proposed above? [1] https://review.opendev.org/#/c/549130/ [2] http://paste.openstack.org/show/781063/ [3] https://github.com/openstack/cinder/blob/410791580ef60ddb03104bf20766859ed9… [4] https://review.opendev.org/#/c/685488/ -- Thanks, Matt

5 10