December 2018 - openstack-discuss

[oslo] Problem when use library "oslo.messaging" for HA Openstack
by Thành Nguyễn Bá 04 Jan '19

04 Jan '19

Dear all, I have a problem when use 'notification listener' oslo-message for HA Openstack. It raise 'oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 172.16.4.125:5672 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'false' but current is 'true''. How can i fix this?. I think settings default in my program set 'durable' is False so it can't listen RabbitMQ Openstack? This is my nova.conf > http://paste.openstack.org/show/738813/ And section [oslo_messaging_rabbit] [oslo_messaging_rabbit] > rabbit_ha_queues = true > rabbit_retry_interval = 1 > rabbit_retry_backoff = 2 > amqp_durable_queues= true *Nguyễn Bá Thành* *Mobile*: 0128 748 0391 *Email*: bathanhtlu(a)gmail.com

3 4

[nova][dev] tracking spec reviews ahead of spec freeze
by melanie witt 04 Jan '19

04 Jan '19

Hey all, Spec freeze is coming up shortly after the holidays on January 10, 2019. Since we don't have much time after returning to work next year before spec freeze, let's keep track of specs that are close to approval or otherwise candidates to focus on in the last stretch before freeze. Here's an etherpad where I've collected a list of possible candidates for focus the first week of January. Feel free to add notes and specs I might have missed: https://etherpad.openstack.org/p/nova-stein-blueprint-spec-freeze Cheers, -melanie

2 3

[nova][ops] Trying to get per-instance live migration timeout action spec unstuck
by Matt Riedemann 04 Jan '19

04 Jan '19

I'm looking at this (previously approved [1]) spec again [2] and trying to sort out what needs to happen to reach agreement on this feature. Note the dependent blueprint is now complete in Stein [3]. The idea is pretty simple: provide new parameters to the live migration API to (1) override [libvirt]/live_migration_completion_timeout [4] and/or (2) provide a timeout action in case the provided (or configured) timeout is reached which would override [libvirt]/live_migration_timeout_action [5]. The use case is also pretty simple: you can have a default timeout and action (abort) configured but there could be cases where you need to override that on a per-instance basis to move a set of VMs off a host for maintenance, so you want to tell nova to force complete (post-copy or pause) in case of a timeout. The abort and force-complete actions are the same as in the API ([6] and [7] respectively). There are two main sticking points against this in the review: 1. This can already be done using existing APIs (as noted) client-side if monitoring the live migration and it times out for whatever you consider a reasonable timeout at the time. 2. The libvirt driver is the only one that currently supports abort and force-complete. For #1, while valid as a workaround, is less than ideal since it would mean having to orchestrate that into any tooling that needs that kind of workaround, be that OSC, openstacksdk, python-novaclient, gophercloud, etc. I think it would be relatively simple to pass those parameters through with the live migration request down to nova-compute and have the parameters override the config options and then it's natively supported in the API. For #2, while also true, I think is not a great reason *not* to support per-instance timeouts/actions in the API when we already have existing APIs that do the same thing and have the same backend compute driver limitations. To ease this, I think we can sort out two things: a) Can other virt drivers that support live migration (xenapi, hyperv, vmware in tree, and powervm out of tree) also support abort and force-complete actions? John Garbutt at least thought it should be possible for xenapi at the Stein PTG. I don't know about the others - driver maintainers please speak up here. The next challenge would be getting driver maintainers to actually add that feature parity, but that need not be a priority for Stein as long as we know it's possible to add the support eventually. b) There are pre-live migration checks that happen on the source compute before we initiate the actual guest transfer. If a user (admin) specified these new parameters and the driver does not support them, we could fail the live migration early. This wouldn't change the instance status but the migration would fail and an instance action event would be recorded to explain why it didn't work, and then the admin can retry without those parameters. This would shield us from exposing something in the API that could give a false sense of functionality when the backend doesn't support it. Given all of this, are these reasonable compromises to continue trying to drive this feature forward, and more importantly, are other operators looking to see this functionality added to nova? Huawei public cloud operators want it because they routinely are doing live migrations as part of maintenance activities and want to be able to control these values per-instance. I assume there are other deployments that would like the same. If this is something you'd like to see move forward, please speak up soon since the nova spec freeze for Stein is January 10. [1] https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/live-m… [2] https://review.openstack.org/#/c/600613/ [3] https://blueprints.launchpad.net/nova/+spec/live-migration-force-after-time… [4] https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.li… [5] https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.li… [6] https://developer.openstack.org/api-ref/compute/?expanded=#delete-abort-mig… [7] https://developer.openstack.org/api-ref/compute/?expanded=#force-migration-… -- Thanks, Matt

5 9

[glance] (Silly) question about container_format and disk_format
by Andre Florath 03 Jan '19

03 Jan '19

Hello! I do not completely understand the parameters 'container_format' and 'disk_format' as described in [1]. The documentation always uses 'the format' but IMHO there might be two formats involved. Are those formats either (A) the formats of the image file that is passed in. Like (from the official documentation [2]) $ openstack image create --disk-format qcow2 --container-format bare \ --public --file ./centos63.qcow2 centos63-image qcow2 / bare are the formats of the passed in image. or (B) the formats that are used internally to store the image Like $ openstack image create --disk-format vmdk --container-format ova \ --public --file ./centos63.qcow2 centos63-image vmdk / ova are formats that are used internally in OpenStack glance to store the image. In this case there must be an auto-detection of the image file format that is passed in and an automatic conversion into the new format. Kind regards Andre [1] https://developer.openstack.org/api-ref/image/v2/index.html?expanded=create… [2] https://docs.openstack.org/glance/pike/admin/manage-images.html

2 2

Re: [Infra][Cinder] Request for voting permission from LINBIT LINSTOR CI (linbit_ci@linbit.com) for Cinder projects
by Woojay Poynter 03 Jan '19

03 Jan '19

Hello, I would like to request a voting permission on LINBIT LINSTOR CI for Cinder project. We are looking to satisfy third-party testing requirement for Cinder volume driver for LINSTOR (https://review.openstack.org/#/c/624233/) The CI's test result of the lastest patch is at http://us.linbit.com:8080/CI-LINSTOR/33/456/ While the driver itself is still being tweaked on the remaining two failed tests, I hope I can have the CI portion of the requirement ready w/ this request. The testbeds are jenkins slave running fresh installation of devstack w/ target patch triggered from gerrit with lvmthin storage backend managed by LINSTOR driver for Cinder. LINSTOR is storage provisioner using DRBD for block replication. Some of the test results of the LINBIT LINSTOR CI running on openstack-dev/ci-sandbox are here; https://review.openstack.org/#/c/627599/ https://review.openstack.org/#/c/627596/ https://review.openstack.org/#/c/626625/ Thank you for your time during the holidays. Sincerely, Woojay Poynter -- 541-435-1001 woojay(a)linbit.com LINBIT | Keeping the Digital World Running DRBD – Corosync – Pacemaker DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. On Fri, Dec 28, 2018 at 1:51 PM Woojay Poynter <woojay(a)linbit.com> wrote: > Hello, > > I would like to request a voting permission on LINBIT LINSTOR CI for > Cinder project. We are looking to satisfy third-party testing requirement > for Cinder volume driver for LINSTOR ( > https://review.openstack.org/#/c/624233/) The CI's test result of the > lastest patch is at http://us.linbit.com:8080/CI-LINSTOR/33/456/ > > While the driver itself is still being tweaked on the remaining two failed > tests, I hope I can have the CI portion of the requirement ready w/ this > request. > > The testbeds are jenkins slave running fresh installation of devstack w/ > target patch triggered from gerrit with lvmthin storage backend managed by > LINSTOR driver for Cinder. LINSTOR is storage provisioner using DRBD for > block replication. > > Some of the test results of the LINBIT LINSTOR CI running on > openstack-dev/ci-sandbox are here; > https://review.openstack.org/#/c/627599/ > https://review.openstack.org/#/c/627596/ > https://review.openstack.org/#/c/626625/ > > Thank you for your time during the holidays. > > Sincerely, > > Woojay Poynter > -- > 541-435-1001 > woojay(a)linbit.com > > LINBIT | Keeping the Digital World Running > DRBD – Corosync – Pacemaker > > DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. > -- Woojay Poynter – Software Developer +1 503 573-1262 x211 woojay(a)linbit.com LINBIT | Keeping the Digital World Running DRBD – Corosync – Pacemaker DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

2 1

[cinder] Is the cinder Active-Active feature OK？
by Jaze Lee 03 Jan '19

03 Jan '19

Hello, In my opinion, all rest api will get to manager of cinder volume. If the manager is using DLM, we can say cinder volume can support active-active. So, can we rewrite the comments and option's help in page https://github.com/openstack/cinder/blob/master/cinder/cmd/volume.py#L78 ? Thanks a lot. -- 谦谦君子

2 2

[neutron] Bug deputy report 17th~23rd Dec 2018
by Paweł Suder 03 Jan '19

03 Jan '19

Hello Neutrons, Following bugs/RFE/issues have been raised during last week. Some of them were already recognized, triaged, checked: >From oldest: https://bugs.launchpad.net/neutron/+bug/1808731 [RFE] Needs to restart metadata proxy with the start/restart of l3/dhcp agent - IMO need to discus how to do that. https://bugs.launchpad.net/neutron/+bug/1808916 Update mailinglist from dev to discuss - it is done. https://bugs.launchpad.net/neutron/+bug/1808917 RetryRequest shouldn't log stack trace by default, or it should be configurable by the exception - confirmed by Sławek ;) https://bugs.launchpad.net/neutron/+bug/1809037 [RFE] Add anti_affinity_group to binding:profile - RFE connected with another RFE from Nova related to NFV and (anti)affinity of resources like PFs. https://bugs.launchpad.net/neutron/+bug/1809080 reload_cfg doesn't work correctly - change in review, I tried to review, but no comments left - new contributor https://bugs.launchpad.net/neutron/+bug/1809134 - TypeError in QoS gateway_ip code in l3-agent logs - review in progress https://bugs.launchpad.net/neutron/+bug/1809238 - [l3] `port_forwarding` cannot be set before l3 `router` in service_plugins - review in progress https://bugs.launchpad.net/neutron/+bug/1809447 - performance regression from mitaka to ocata - not triaged, I am not sure how to handle it - it is a wide thing.. https://bugs.launchpad.net/neutron/+bug/1809497 - bug noticed on gates related to another bug opened last week: https://bugs.launchpad.net/neu tron/+bug/1809134 Wish you a good time this week and for the next year! :) Cheers, Paweł

2 2

[all][ptl][heat][senlin][magnum]New SIG for Autoscaling? plus Session Summary: Autoscaling Integration, improvement, and feedback
by Rico Lin 03 Jan '19

03 Jan '19

Dear all Tl;dr; I gonna use this ML to give a summary of the forum [1] and asking for feedback for the idea of new SIG. We have a plan to create a new SIG for autoscaling which to cover the common library, docs, and tests for cross-project services (Senlin, Heat, Monasca, etc.) and cross-community (OpenStack, Kubernetes, etc). And the goal is not just to have a place to keep those resources that make sure we can guarantee the availability for use cases, also to have a force to keep push the effort to integrate across services or at least make sure we don't go to a point where everyone just do their own service and don't care about any duplication. So if you have any thoughts for the new SIG (good or bad) please share it here. Here I summarize our discussion in the forum session `Autoscaling Integration, improvement, and feedback`. If you like to learn more or input your thoughts, feel free to put it in etherpad [1] or simply reply to this email. In the forum, we have been discussed the scope and possibility to integrate effort from Heat, Senlin, and also autoscaling across OpenStack to K8s. There are some long-term goals that we can start on like create a document for general Autoscaling on OpenStack, Common library for cross-project usage, or create real scenario cases to test on our CI. And the most important part is how can we help users and satisfied use cases without confuse them or making too much duplication effort across communities/projects. So here's an action we agree on, is to trigger discussion for either we need to create a new SIG for autoscaling. We need to define the scope and the goal for this new SIG before we go ahead and create one. The new SIG will cover the common library, docs, and tests for cross-project services (Senlin, Heat, Monasca, etc.) and cross-community (OpenStack, Kubernetes, etc). And the goal is not just to have a place to keep those resources that make sure we can guarantee the availability for use cases, also to have a force to keep push the effort to integrate across services or at least make sure we don't go to a point where everyone just do their own service and don't care about any duplication. For example, we can have a document about do autoscaling in OpenStack, but we need a place to put it and keep maintain it. And we can even have a place to set up CI to test all scenario for autoscaling. I think it's possible to extend the definition of this SIG, but we have to clear our goal and make sure we actually doing a good thing and make everyone's life easier. On the other hand we also need to make sure we do not duplicate the effort of other SIGs/WGs. Also The reason I add `ptl` tag for this ML is that this SIG or the concept of `autoscaling` might be very deferent to different projects. So I really wish to hear from anyone and any projects who are interested in this topic. [1] https://etherpad.openstack.org/p/autoscaling-integration-and-feedback -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

4 3

[nova] Granular locks in the API
by Matt Riedemann 02 Jan '19

02 Jan '19

I wanted to float something that we talked about in the public cloud SIG meeting today [1] which is the concept of making the lock API more granular to lock on a list of actions rather than globally locking all actions that can be performed on a server. The primary use case we discussed was around a pre-paid pricing model for servers. A user can pre-pay resources at a discount if let's say they are going to use them for a month at a fixed rate. However, once they do, they can't resize those servers without going through some kind of approval (billing) process to resize up. With this, the provider could lock the user from performing the resize action on the server but the user could do other things like stop/start/reboot/snapshot/etc. The pricing model sounds similar to pre-emptible instances for getting a discount but the scenario is different in that these servers couldn't be pre-empted (they are definitely more non-cloudy pets than cattle). An alternative solution for that locked resize issue is using granular policy rules such that pre-paid servers have some other kind of role attached to them so by policy you could restrict users from performing actions on those servers (but the admin could override). In reality I'm not sure how feasible that is in a public cloud with several thousand projects. The issue I see with policy controlling this is the role is attached to the project, not the resource (the server), so if you did this would users have to have separate projects for on-demand vs pre-paid resources? I believe that's what CERN and StackHPC are doing with pre-emptible instances (you have different projects with different quota models for pre-emptible resources). I believe there are probably other use cases for granular locks on servers for things like service VMs (trove creates some service VMs to run a database cluster and puts locks on those servers). Again, definitely a pet scenario but it's one I've heard before. Would people be generally in favor of this or opposed, or just meh? [1] https://etherpad.openstack.org/p/publiccloud-wg -- Thanks, Matt

4 5

queens heat db deadlock
by Ignazio Cassano 02 Jan '19

02 Jan '19

Hi All, I installed queens on centos 7. Heat seems to work fine with templates I wrote but when I create magnum cluster I often face with db deadlock in heat-engine log: 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource Traceback (most recent call last): 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 918, in _action_recorder 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource yield 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2035, in delete 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource *action_args) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource step = next(subtask) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 977, in action_handler_task 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource done = check(handler_data) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 587, in check_delete_complete 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource return self._check_status_complete(self.DELETE) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 454, in _check_status_complete 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource action=action) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource ResourceFailure: resources[0]: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') (Background on this error at: http://sqlalche.me/e/2j85) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource 2018-12-13 11:48:46.030 89597 INFO heat.engine.stack [req-9a43184f-77d8-4fad-ab9e-2b9826c10b70 - admin - default default] Stack DELETE FAILED (swarm-clustergp27-ebhsalhb4bop-swarm_primary_master-tzgkh3ncmymw): Resource DELETE failed: resources[0]: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') (Background on this error at: http://sqlalche.me/e/2j85) 2018-12-13 11:48:46.844 89595 INFO heat.engine.resource [req-9a43184f-77d8-4fad-ab9e-2b9826c10b70 - admin - default default] DELETE: ResourceGroup "swarm_primary_master" [6984db3c-3ac1-4afc-901f-21b3e7f230a7] Stack "swarm-clustergp27-ebhsalhb4bop" [b43256fa-52e2-4613-ac15-63fd9340a8be] I read this issue was solved in heat version 10 but seems not. Regards Ignazio

5 14