- openstack-discuss - lists.openstack.org

[nova] instence running, but status is Error
by Budai Laszlo 26 Jul '19

26 Jul '19

Dear all, I am testing different migration scenarios in our openstack, and I ended up with an instance that is running (I can access it by ssh), but its Status is Error. and its task_state is resize_migrating. I suppose that I got here due to the fact that the `allow_resize_to_same_host` is set to True, and the scheduler has tried to use the same compute node in the migration as described here: https://bugs.launchpad.net/nova/+bug/1811235. Is there any possibility to be able to manage again that instance using openstack (except to delete it)? I have tried the shut off (from horizon) and it gives me an error: fault: code: 400 created: '2019-07-26T14:46:05Z' message: Unavailable console type novnc. Thank you, Laszlo

2 3

[placement] update 19-29
by Chris Dent 26 Jul '19

26 Jul '19

HTML: https://anticdent.org/placement-update-19-29.html Welcome to a rushed pupdate 19-29. My morning was consumed by other things. A reminder: The Placement project holds office hours every Wednesday at 1500 UTC in the `#openstack-placement` IRC channel. If you have a topic that needs some synchronous discussion, then is an ideal time. Just start talking! # Most Important The two main things on the Placement radar are implementing Consumer Types and cleanups, performance analysis, and documentation related to nested resource providers. # What's Changed * The [api-ref](https://docs.openstack.org/api-ref/placement/) has moved to `docs.openstack.org` from `developer.openstack.org`. Redirects are in place. * Both traits and resource classes are [now cached](https://review.opendev.org/672298) per request, allowing for some name to id and id to name optimizations. * A new zuul template is [being used](https://review.opendev.org/671257) in placement that means fewer irrelevant tempest tests are run on placement changes. # Stories/Bugs (Numbers in () are the change since the last pupdate.) There are 21 (-1) stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 (0) are [untagged](https://storyboard.openstack.org/#!/worklist/580). 2 (0) are [bugs](https://storyboard.openstack.org/#!/worklist/574). 5 (0) are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 10 (0) are [rfes](https://storyboard.openstack.org/#!/worklist/594). 4 (0) are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 17 (0). * Placement related nova [in progress bugs](https://goo.gl/vzGGDQ) on launchpad: 4 (0). # osc-placement osc-placement is currently behind by 12 microversions. * <https://review.opendev.org/666542> Add support for multiple member_of. There's been some useful discussion about how to achieve this, and a consensus has emerged on how to get the best results. # Main Themes ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. * <https://review.opendev.org/#/q/topic:bp/support-consumer-types> A WIP, as microversion 1.37, has started. ## Cleanup Cleanup is an overarching theme related to improving documentation, performance and the maintainability of the code. The changes we are making this cycle are fairly complex to use and are fairly complex to write, so it is good that we're going to have plenty of time to clean and clarify all these things. # Other Placement Miscellaneous changes can be found in [the usual place](https://review.opendev.org/#/q/project:openstack/placement+status:op…. There are two [os-traits changes](https://review.opendev.org/#/q/project:openstack/os-traits+status:… being discussed. And zero [os-resource-classes changes](https://review.opendev.org/#/q/project:openstack/os-resource-class… (yay!). # Other Service Users New discoveries are added to the end. Merged stuff is removed. Anything that has had no activity in 4 weeks has been removed. * <https://review.openstack.org/#/q/topic:bug/1819923> Nova: nova-manage: heal port allocations * <https://review.opendev.org/659233> Cyborg: Placement report * <https://review.opendev.org/662229> helm: add placement chart * <https://review.opendev.org/656023> Nova: Use OpenStack SDK for placement * <https://review.opendev.org/634551> libvirt: report pmem namespaces resources by provider tree * <https://review.opendev.org/660852> Nova: Remove PlacementAPIConnectFailure handling from AggregateAPI * <https://review.opendev.org/#/q/topic:bp/support-move-ops-with-qos-ports> Nova: support move ops with qos ports * <https://review.opendev.org/664689> Nova: get_ksa_adapter: nix by-service-type confgrp hack * <https://review.opendev.org/666202> Blazar: Create placement client for each request * <https://review.opendev.org/667952> nova: Support filtering of hosts by forbidden aggregates * <https://review.opendev.org/669079> blazar: Send global_request_id for tracing calls * <https://review.opendev.org/668252> Nova: Update HostState.\*\_allocation_ratio earlier * <https://review.opendev.org/670112> Nova: WIP: Add a placement audit command * <https://review.opendev.org/586960> zun: [WIP] Use placement for unified resource management * <https://review.opendev.org/671478> Kayobe: Build placement images by default * <https://review.opendev.org/671312> blazar: Fix placement operations in multi-region deployments * <https://review.opendev.org/672870> Watcher: Getting data from placement when updating datamodel * <https://review.opendev.org/671793> Nova: libvirt: Start reporting PCPU inventory to placement * <https://review.opendev.org/672649> chef: placement_api: create valid apache config before installing package * <https://review.opendev.org/670696> tempest: Add placement API methods for testing routed provider nets * <https://review.opendev.org/672678> openstack-helm: Build placement in OSH-images # End If we get the chance, it will be interesting to start working with placement with 1 million providers. Just to see. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent

1 0

[openstack-ansible] weekly update
by Mohammed Naser 26 Jul '19

26 Jul '19

Hi everyone, I'm going to try doing these more often to keep everyone updated on the current state of openstack-ansible (mostly around what is discussed during our office hours / meetings). - Dmitriy has been working on removing unused code in playbooks, mostly around logs as we move to 100% systemd-journald. - We're discussing the proposal to remove os-log-dir-setup.yml, as well as rsyslog_client and rsyslog_server roles as they aren’t used anymore with the move to systemd-journald - We're also discussing removing all the complex bind mounts, which would clean it and simplify the code base. - Our continuous integration has been stable but needs upgrade jobs and increased coverage. Our coverage would go up significantly by running aio_lxc for all supported systems. - Jonathan's work on binding to management IPs is making progress (and seems to be at the Galera stage). - CentOS CI has been very slow and nobody knows why. It seems to be something we've regressed at some point. - A unicast flood issue affecting deployments using linuxbridge that happened inside os-vif needs to be patched. It's still pending os-vif to patch. - Namrata mentioned an issue during the upgrade from Rocky to Stein (when upgrading WSREP SST method from xtrabackup-V2 to mariabackup) that will have to be fixed is Stein. That's something that's in progress now. Thanks everyone for reading! Regards, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

2 1

[cinder][ceph][RBD mirroring] Error while creating a mirrored volume
by Ajitha Robert 26 Jul '19

26 Jul '19

Hi all Following picture shows my setup [image: image.png] I m facing following error when i issue a command to enable rbd mirroring on a particular image either manually or through cinder. *"rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x7fb364025310 finish: resending after timeout"* in rbd-mirroring.log and getting *"failed to get omap key: client_0921b3ce-716a-4462-b3cf-86b5102bfd9"* error in osd.log. . Should I enable anything else apart from timeout parametes, if the distance between the two ceph clusters is greater than 300km? when I create a bootable volume of 100 GB with a glance image.Image get downloaded and from cinder, volume log throws with "volume is busy deleting volume that has snapshot" . Image was enabled with exclusive lock, journaling, layering, object-map, fast-diff and deep-flatten Cinder volume is in error state but the rbd image is created in primary but not in secondary. pls check the logs 1)Log for manual rbd image creation http://paste.openstack.org/show/754766/ 2)Log for 16gb volume created from cinder, status in cinder volume is available http://paste.openstack.org/show/754767/ 3)Log for 100gb volume created from cinder, status in cinder volume is error http://paste.openstack.org/show/754769/ -- *Regards,Ajitha R*

1 0

[placement] update 19-27
by Chris Dent 25 Jul '19

25 Jul '19

HTML: https://anticdent.org/placement-update-19-27.html Pupdate 19-27 is here and now. # Most Important Of the features we planned to do this cycle, all are done save one: consumer types (in progress, see below). This means we have a good opportunity to focus on documentation, performance, and improving the codebase for maintainability. You do not need permission to work on these things. If you find a problem and know how to fix it, fix it. If you are not sure about the solution, please discuss it on this email list or in the `#openstack-placement` IRC channel. This also means we're in a good position to help review changes that use placement in other projects. The Foundation needs to know how much, if any, Placement time will be needed in Shanghai. I started a [thread](http://lists.openstack.org/pipermail/openstack-discuss/2019-July/00… and an [etherpad](https://etherpad.openstack.org/p/placement-shanghai-ptg). # What's Changed * The `same_subtree` query parameter has merged as [microversion 1.36](https://docs.openstack.org/placement/latest/placement-api-microversio…. This enables a form of nested provider affinity: "these providers must all share the same ancestor". * The placement projects have been updated (pending merge) to match the [Python 3 test runtimes for Train](https://governance.openstack.org/tc/goals/train/python3-updates.html) community wide goal. Since the projects have been Python3-enabled from the start, this was mostly a matter of aligning configurations with community-wide norms. When Python 3.8 becomes available we should get on that sooner than later to catch issues early. # Specs/Features All placement specs have merged. Thanks to everyone for the frequent reviews and quick followups. Some non-placement specs are listed in the Other section below. # Stories/Bugs (Numbers in () are the change since the last pupdate.) There are 23 (0) stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 (0) are [untagged](https://storyboard.openstack.org/#!/worklist/580). 3 (1) are [bugs](https://storyboard.openstack.org/#!/worklist/574). 5 (0) are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 11 (0) are [rfes](https://storyboard.openstack.org/#!/worklist/594). 4 (0) are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 16 (0). * Placement related nova [in progress bugs](https://goo.gl/vzGGDQ) on launchpad: 4 (0). # osc-placement osc-placement is currently behind by 11 microversions. * <https://review.opendev.org/666542> Add support for multiple member_of. This is stuck and needs input from users of the tool. # Main Themes ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. * <https://review.opendev.org/#/q/topic:bp/support-consumer-types> This is currently the migration to add the necessary table and column. ## Cleanup Cleanup is an overarching theme related to improving documentation, performance and the maintainability of the code. The changes we are making this cycle are fairly complex to use and are fairly complex to write, so it is good that we're going to have plenty of time to clean and clarify all these things. As mentioned last week, one of the important cleanup tasks that is not yet in progress is updating the [gabbit](https://opendev.org/openstack/placement/src/branch/master/gate/gabb… that creates the nested topology that's used in nested performance testing. The topology there is simple, unrealistic, and doesn't sufficiently exercise the several features that may be used during a query that desires a nested response. This needs to be someone who is more closely related to real world use of nested than me. efried? gibi? Another cleanup that needs to start is satisfying the community wide goal of [PDF doc generation](https://storyboard.openstack.org/#!/story/2006110). Anyone know if there is a cookbook for this? # Other Placement Miscellaneous changes can be found in [the usual place](https://review.opendev.org/#/q/project:openstack/placement+status:op…. There are three [os-traits changes](https://review.opendev.org/#/q/project:openstack/os-traits+status:… being discussed. And one [os-resource-classes change](https://review.opendev.org/#/q/project:openstack/os-resource-classe…. # Other Service Users New discoveries are added to the end. Merged stuff is removed. Anything that has had no activity in 4 weeks has been removed. * <https://review.openstack.org/#/q/topic:bug/1819923> Nova: nova-manage: heal port allocations * <https://review.openstack.org/650188> nova-spec: Allow compute nodes to use DISK_GB from shared storage RP * <https://review.opendev.org/659233> Cyborg: Placement report * <https://review.opendev.org/662229> helm: add placement chart * <https://review.opendev.org/656023> Nova: Use OpenStack SDK for placement * <https://review.opendev.org/612497> Nova: Spec: Provider config YAML file * <https://review.opendev.org/634551> libvirt: report pmem namespaces resources by provider tree * <https://review.opendev.org/660852> Nova: Remove PlacementAPIConnectFailure handling from AggregateAPI * <https://review.opendev.org/#/q/topic:bp/support-move-ops-with-qos-ports> Nova: support move ops with qos ports * <https://review.opendev.org/664689> Nova: get_ksa_adapter: nix by-service-type confgrp hack * <https://review.opendev.org/664862> OSA: Add nova placement to placement migration * <https://review.opendev.org/657796> Nova: Defaults missing group_policy to 'none' * <https://review.opendev.org/666202> Blazar: Create placement client for each request * <https://review.opendev.org/669309> tempest: Define the Integrated-gate-placement gate template * <https://review.opendev.org/668263> Nova: Restore RT.old_resources if ComputeNode.save() fails * <https://review.opendev.org/667952> nova: Support filtering of hosts by forbidden aggregates * <https://review.opendev.org/669079> blazar: Send global_request_id for tracing calls * <https://review.opendev.org/668252> Nova: Update HostState.\*\_allocation_ratio earlier * <https://review.opendev.org/670105> neutron: segments: fix rp inventory update * <https://review.opendev.org/670112> Nova: WIP: Add a placement audit command * <https://review.opendev.org/668526> OSA: Add placement client for neutron * <https://review.opendev.org/586960> zun: [WIP] Use placement for unified resource management # End A colleague suggested yesterday that the universe doesn't have an over subscription problem, rather there's localized contention, and what we really have is a placement problem. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent

2 1

[keystone] Need clarification about Stein policies
by Bernd Bausch 25 Jul '19

25 Jul '19

The Keystone policy.json file I created with oslo-policy-generator contains lines I don't understand. For example /list_users/. The comment says: # DEPRECATED "identity:list_users":"rule:admin_required" has been # deprecated since S in favor of "identity:list_users":"(role:reader # and system_scope:all) or (role:reader and # domain_id:%(target.domain_id)s)". I do understand the expression starting with (role:reader .... , but contrarily to the comment, the policy is "identity:list_users": "rule:identity:list_users" This looks like a circular definition, and in any case, nowhere do I seerule:identity:list_users defined. Can someone in the know explain how this policy is processed? Thanks much, Bernd

2 1

[dev][nova][glance][cinder] introducing 'compressed' image container_format
by Ma, Zheng 25 Jul '19

25 Jul '19

Hello Nova developers, Cinder is working on a feature that will allow it to compress volumes uploaded to Glance [0,1]. We are proposing a new container_format for Glance named 'compressed' [2]. The idea is that a consumer of a 'compressed' image would read the file to determine the kind of compression and take appropriate action (either successfully decompress to find a payload that matches the image's disk_format, or fail with 'unsupported container_format'). Our preliminary testing with a raw image with container_format == compressed, disk_format == raw is that Nova boots a VM from the image but the VM appears to be unusable. Nova seems taking it as a real RAW image as disk_format indicates. So we want to open a discussion about handling this. Thanks! [0] cinder spec patch https://review.opendev.org/#/c/652275/ [1] cinder code implementation https://review.opendev.org/#/c/668825 [2] glance spec patch https://review.opendev.org/#/c/670454/

1 0

[Nova] migration issue
by Budai Laszlo 25 Jul '19

25 Jul '19

Dear all, we are testing the cold migratoin of instances that are using local ephemeral storage and it fails with the following error: 2019-07-24 13:47:13.115 58902 INFO nova.virt.libvirt.driver [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae 58-00d6-4317-b635-909fdf09ac49] Instance shutdown successfully after 3 seconds. 2019-07-24 13:47:13.126 58902 INFO nova.virt.libvirt.driver [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Instance destroyed successfully. 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58- 00d6-4317-b635-909fdf09ac49] Setting instance vm_state to ERROR: ProcessExecutionError: Unexpected error while running command. Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph 0 Exit code: 1 Stdout: u'' Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured syst em and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. * \n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No suc h file or directory\n' 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Traceback (most recent call last): 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 75 18, in _error_out_instance_on_exception 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] yield 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 42 75, in _resize_instance 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] timeout, retry_interval) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin e 8200, in migrate_disk_and_power_off 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] shared_storage) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220 , in __exit__ 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] self.force_reraise() 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196 , in force_reraise 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin e 8185, in migrate_disk_and_power_off 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in copy_image 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs .py", line 110, in copy_file 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs .py", line 196, in copy_file 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] on_execute=on_execute, on_completion=on_completion) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in exec ute 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] return processutils.execute(*cmd, **kwargs) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py" , line 424, in execute 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] cmd=sanitized_cmd) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] ProcessExecutionError: Unexpected error while running command. 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Exit code: 1 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stdout: u'' 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] 2019-07-24 13:47:14.380 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Swapping old allocation on 3da11b23-5ef8-4471-a3c9-5f743b0bbfd7 held by migration da66c416-2636-421b-99cb-470bd07dad4c for instance 2019-07-24 13:47:14.979 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Successfully reverted task state from resize_migrating on failure for instance. 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] Exception during message handling: ProcessExecutionError: Unexpected error while running command. Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 Exit code: 1 Stdout: u'' Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server function_name, call_dict, binary) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4240, in resize_instance 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self._revert_allocation(context, instance, migration) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4237, in resize_instance 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server instance_type, clean_shutdown) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4275, in _resize_instance 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server timeout, retry_interval) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8200, in migrate_disk_and_power_off 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server shared_storage) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8185, in migrate_disk_and_power_off 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server compression=compression) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in copy_image 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server compression=compression) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in copy_file 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server compression=compression) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in copy_file 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server on_execute=on_execute, on_completion=on_completion) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in execute 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command. 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Exit code: 1 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stdout: u'' 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server 2019-07-24 13:47:17.065 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] image ca89475a-cf15-46d1-ab2d-c2b96c403248 at (/var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07): checking 2019-07-24 13:47:17.066 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] Active base files: /var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07 2019-07-24 13:47:27.768 58902 INFO nova.compute.manager [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] VM Stopped (Lifecycle Event) 2019-07-24 13:47:27.918 58902 INFO nova.compute.manager [req-e53e928d-81fe-4185-9e62-abc2113e838c - - - - -] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. The disk has its root FS n an RBD device (a volume created by cinder) and it has its ephemeral disk on a local LVM volume created by nova. It seems that the /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 is not there when scp is trying to copy it to the other side. Any ideas how to fix this? The nova config on the nodes looks like this: # Ansible managed [DEFAULT] allow_resize_to_same_host = True # Compute compute_driver = libvirt.LibvirtDriver # Scheduler cpu_allocation_ratio = 2.0 # Logs / State debug = False # Hypervisor default_ephemeral_format = ext4 disk_allocation_ratio = 1.0 # Api's enabled_apis = osapi_compute,metadata executor_thread_pool_size = 64 fatal_deprecations = False # Configdrive force_config_drive = False host = compute8.mgmt.lab.mydomain.com image_cache_manager_interval = 0 instance_name_template = instance-%08x # Ceilometer notification configurations instance_usage_audit = True instance_usage_audit_period = hour instances_path = /var/lib/nova/instances ## Vif libvirt_vif_type = ethernet log_dir = /var/log/nova # Metadata metadata_workers = 16 # Network my_ip = 192.168.56.47 notify_on_state_change = vm_and_task_state osapi_compute_workers = 16 ram_allocation_ratio = 1.0 reserved_host_disk_mb = 0 reserved_host_memory_mb = 2048 resume_guests_state_on_host_boot = False rootwrap_config = /etc/nova/rootwrap.conf rpc_response_timeout = 60 service_down_time = 120 state_path = /var/lib/nova # Rpc all transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e 8cf8bd7@192.168.56.179:5671//nova # Disable stderr logging use_stderr = False vif_plugging_is_fatal = True vif_plugging_timeout = 30 [api] auth_strategy = keystone enable_instance_password = True use_forwarded_for = False vendordata_jsonfile_path = /etc/nova/vendor_data.json # Cache [cache] backend = oslo_cache.memcache_pool enabled = true memcache_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 # Cinder [cinder] cafile = /etc/ssl/certs/ca-certificates.crt catalog_info = volumev3:cinderv3:internalURL cross_az_attach = True os_region_name = A_Lab [conductor] workers = 16 [filter_scheduler] available_filters = nova.scheduler.filters.all_filters enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, AggregateRamFilter, ComputeFilter, AggregateCoreFilter, DiskFilter, AggregateDiskFilter, AggregateNumInstancesFilter, AggregateIoOpsFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, NUMATopologyFilter, SameHostFilter, DifferentHostFilter host_subset_size = 10 max_instances_per_host = 50 max_io_ops_per_host = 10 ram_weight_multiplier = 5.0 tracks_instance_changes = True weight_classes = nova.scheduler.weights.all_weighers # Glance [glance] api_servers = https://vip.mgmt.lab.mydomain.com:9292 cafile = /etc/ssl/certs/ca-certificates.crt [key_manager] fixed_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [keystone_authtoken] auth_type = password auth_uri = https://vip.mgmt.lab.mydomain.com:5000 auth_url = https://vip.mgmt.lab.mydomain.com:35357 cafile = /etc/ssl/certs/ca-certificates.crt insecure = False memcache_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # if your memcached server is shared, use these settings to avoid cache poisoning memcache_security_strategy = ENCRYPT memcached_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 password = xxxxxxxxxxxxxxxxxxxxxxx project_domain_id = default project_name = service region_name = A_Lab token_cache_time = 300 user_domain_id = default username = nova [libvirt] disk_cachemodes = network=writeback hw_disk_discard = unmap images_rbd_ceph_conf = /etc/ceph/ceph.conf images_rbd_pool = vms images_type = lvm images_volume_group = ssd_nova inject_key = False inject_partition = -2 inject_password = False live_migration_tunnelled = True live_migration_uri = "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa" rbd_secret_uuid = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # ceph rbd support rbd_user = cinder remove_unused_resized_minimum_age_seconds = 3600 use_virtio_for_bridges = True virt_type = kvm # Neutron [neutron] auth_type = password # Keystone client plugin authentication URL option auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 cafile = /etc/ssl/certs/ca-certificates.crt default_floating_pool = public insecure = False metadata_proxy_shared_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxx # Keystone client plugin password option password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx project_domain_name = Default project_name = service region_name = A_Lab service_metadata_proxy = True url = https://vip.mgmt.lab.mydomain.com:9696 user_domain_name = Default # Keystone client plugin username option username = neutron [oslo_concurrency] lock_path = /var/lock/nova # Notifications [oslo_messaging_notifications] driver = messagingv2 notification_topics = notifications transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.179:5671//nova [oslo_messaging_rabbit] rpc_conn_pool_size = 30 ssl = True # Placement [placement] auth_type = "password" auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 cafile = /etc/ssl/certs/ca-certificates.crt insecure = False os_interface = internal os_region_name = A_Lab password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx project_domain_name = Default project_name = service user_domain_name = Default username = placement [quota] cores = 20 injected_file_content_bytes = 10240 injected_file_path_length = 255 injected_files = 5 instances = 10 key_pairs = 100 max_age = 0 metadata_items = 128 ram = 32768 server_group_members = 10 server_groups = 10 [scheduler] discover_hosts_in_cells_interval = 60 host_manager = host_manager max_attempts = 5 periodic_task_interval = 60 scheduler_driver = filter_scheduler [spice] agent_enabled = True enabled = True # Console Url and binds html5proxy_base_url = https://dashboard.iaas.lab.mydomain.com:6080/spice_auto.html server_listen = 192.168.56.47 server_proxyclient_address = 192.168.56.47 [upgrade_levels] compute = auto [vnc] enabled = False [wsgi] api_paste_config = /etc/nova/api-paste.ini secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO Thank you in advance for any suggestions. Kind regards, Laszlo

5 10

[nova] Train Spec Freeze
by Eric Fried 25 Jul '19

25 Jul '19

Hi Nova contributors and maintainers. Spec freeze for Nova is Thursday, July 25th. This means that any specs not approved by EOD [1] become drastically less likely to land in Train. For your convenience, here is a search that's intended to catch open Train specs while filtering out things like updates to existing specs: https://review.opendev.org/#/q/project:openstack/nova-specs+status:open+pat… At the time of this writing, there are 15 on that list. Of those, IMO two are "close": Add Unified Limits Spec - https://review.opendev.org/602201 - Two +1s, needs review from "Nova API subteam" Spec for the Nova part of Image Encryption - https://review.opendev.org/608696 - +3, needs a second core If you own a spec you think should be able to land [2], please: - Check it to make sure any comments have been addressed (even if there's no -1). - Explicitly ask for reviews, here on the mailing list and/or in IRC in #openstack-nova. Reviewers, please make an effort to ensure that all the specs on that list have at least one vote. (Currently there are six with no vote.) I'll send an update Friday or Monday with a final tally. Thanks, efried [1] i.e. by the time cores have signed off for the day, wherever they are [2] meaning it has already been through some reviews and seems to have support. If you just proposed your spec this week, it's not reasonable to expect it to land.

2 1

Heat resource types - aodh and ceilometer
by Dhairyasheel Ghadge 25 Jul '19

25 Jul '19

Hello all, I have installed heat engine along with heat dashboard for orchestration in openstack cloud. I am creating a template to autoscale a web server pool and spin a new VM when load on other one increases. For the autoscaling purpose, I require to have an alarming system which will monitor the cp meter of the instance, which is provided by ceilometer and aodh services of openstack. I cannot find resource types of aodh and ceilometer on the heat dashboard. These same resource types are present in the heat openstack resources directory. Are there any steps to be carried out to make it available on the dashboard? Thanks & Regards Dhairyasheel

1 0

[nova] [horizon] Horizon and nova-cli inconsistency in live migration parameters
by Mary Atakova 25 Jul '19

25 Jul '19

Hi everyone, I think I have found an inconsistency between nova cli and Horizon in the way they send live migration commands to nova API. I'm not sure where this needs to be fixed, so please advise on that. The situation: I try to live-migrate a VM, and I do it through nova cli and through Horizon, and nova-api logs contain different data depending on the way I do it. 1. nova cli I run the command: nova live-migration 17359460-d23c-4acc-a9b1-5cf117b54430 and it shows up in nova-api logs as: {"os-migrateLive": {"block_migration": "auto", "host": null}} 2. horizon I run live migration via Horizon with unchecked Disk Overcommit and unchecked Block Migration. It shows up in nova-api logs as: {"os-migrateLive": {"disk_over_commit": false, "block_migration": false, "host": null}} This difference can lead to different parameters provided for live migration when block_migration: auto is translated into block_migration: true. My environment: Openstack Stein CentOS Linux release 7.6.1810 nova versions: # yum list installed | grep nova openstack-nova-api.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-common.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-conductor.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-console.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-novncproxy.noarch openstack-nova-placement-api.noarch openstack-nova-scheduler.noarch 1:19.0.1-1.el7 @centos-openstack-stein python2-nova.noarch 1:19.0.1-1.el7 @centos-openstack-stein python2-novaclient.noarch 1:13.0.0-1.el7 @centos-openstack-stein horizon version: # yum list installed | grep dashboard openstack-dashboard.noarch 1:15.1.0-1.el7 @centos-openstack-stein openstack-dashboard-theme.noarch 1:15.1.0-1.el7 @centos-openstack-stein Thank you in advance for your attention Best regards, Mary

2 2

DuplicateMessageError after restart of a control node
by Pawel Konczalski 25 Jul '19

25 Jul '19

Hello everybody, after restart of one of the control nodes i see this error message every few seconds int the logs: @timestamp July 23rd 2019, 16:44:08.927 t Hostname ctlX t Logger openstack.nova t Payload Failed to process message ... skipping it.: DuplicateMessageError: Found duplicate message(f2178cffb5d249f3a1d2df1c0322f18c). Skipping it. 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 361, in _callback 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit self.callback(RabbitMessage(message)) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 251, in __call__ 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit unique_id = self.msg_id_cache.check_duplicate_message(message) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py", line 122, in check_duplicate_message 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit raise rpc_common.DuplicateMessageError(msg_id=msg_id) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit DuplicateMessageError: Found duplicate message(f2178cffb5d249f3a1d2df1c0322f18c). Skipping it. 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit t Pid 82 t Timestamp 2019-07-23 16:44:08.927 t _id AWwfSjEtsndUu9YeotwF t _index flog-2019.07.23 # _score - t _type fluentd t log_level ERROR t programname nova-scheduler t python_module oslo.messaging._drivers.impl_rabbit As soon as the restarted node is taken out of the cluster, the errors disappear. Any idea how to fix this? Thank you Pawel

2 2

[release] (Belated) Release countdown for week R-12, July 22-26
by Kendall Nelson 24 Jul '19

24 Jul '19

Hello Everyone, Development Focus ----------------- The Train-2 milestone is July 25! Train-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/train/schedule.html for details. General Information ------------------- Libraries need to be released at least once per milestone period. This week, the release team will propose releases for any library that has not been otherwise released since milestone 1. See list at: http://paste.openstack.org/show/754817/. PTL's and release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well. A +1 would be appreciated, but if we do not hear anything at all by the end of the week, we will assume things are OK to proceed. Remember that non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. This week is also the deadline to freeze the contents of the final release. All new 'Train' deliverables need to have a deliverable file ( https://opendev.org/openstack/releases/src/branch/master/deliverables/train) and need to have done a release by milestone-2. The following new deliverables have not had a release yet (or not been already discussed in the etherpad[1]), and will not be included in Train unless a release is requested for them in the coming week: python-adjutantclient, cyborg-tempest-plugin, neutron-interconnection, openstack-helm, openstack-helm-addons, openstack-help-docs, openstack-helm-images, openstack-helm-infra, and compute-hyperv. The full list of currently deliverable-less deliverables can be found here[2]. Upcoming Deadlines & Dates -------------------------- Train-2 Milestone: July 25 (R-12 week) Thanks! -Kendall Nelson(diablo_rojo) + the Release Management Team [1]https://etherpad.openstack.org/p/train-membership-freeze [2] http://paste.openstack.org/show/754816/

1 0

[oslo] PTO next week
by Ben Nemec 24 Jul '19

24 Jul '19

Just a heads up that I will be on PTO next week and incommunicado pretty much the whole time. Moises (moguimar) volunteered to run the Oslo meeting so that will still be happening. Herve (hberaud) and Doug (dhellmann) are our release liaisons so if you need a release of any Oslo libraries next week, they're the ones to talk to. Otherwise, things have been pretty quiet in Oslo-land (which would be Norway, I guess), so I expect all the things to break next week while I'm out. Good luck. ;-) Thanks. -Ben

1 0

[ops] [nova] How to change/remove a property from an instance ?
by Massimo Sgaravatto 24 Jul '19

24 Jul '19

In my OpenStack Rocky cloud I have some instances that were created (before updating to Rocky) using images with the property hypervisor_type='qemu'. Since in my compute nodes [libvirt]/virt_type is "kvm", after having updated to Rocky I had to change these images removing that property of changing its value (qemu --> kvm): this is discussed in the thread: http://lists.openstack.org/pipermail/openstack-discuss/2019-July/thread.htm… Now I am not able to migrate anymore the instances that were created using those images (ImagePropertiesFilter filters out all compute nodes) before fixing the property. As suggested by Matt I tried to play with the instance_system_metadata table of the nova database. I first tried to change the value of that property (qemu --> kvm) [*] I also tried to remove the property [**] but the nova migrate keeps failing. In the request I keep seeing that Instance contains properties ImageMetaProps...img_hv_type='qemu' What else needs to be modified? Thanks, Massimo [*] mysql> select * from instance_system_metadata where instance_uuid='57dcdc2f-ef30-4512-beff-128c1621c5c6'; +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | created_at | updated_at | deleted_at | id | instance_uuid | key | value | deleted | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | 2017-05-15 12:32:57 | NULL | NULL | 9868217 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_hypervisor_type | kvm | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868220 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_disk_format | qcow2 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868223 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_container_format | bare | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868226 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_ram | 512 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868229 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_disk | 20 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868232 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_base_image_ref | c2d3e37c-b14f-4472-9bc2-d75f26ab900d | 0 | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ [**] mysql> delete from instance_system_metadata where id='9868217'; Query OK, 1 row affected (0.01 sec) mysql> select * from instance_system_metadata where instance_uuid='57dcdc2f-ef30-4512-beff-128c1621c5c6'; +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | created_at | updated_at | deleted_at | id | instance_uuid | key | value | deleted | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | 2017-05-15 12:32:57 | NULL | NULL | 9868220 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_disk_format | qcow2 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868223 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_container_format | bare | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868226 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_ram | 512 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868229 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_disk | 20 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868232 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_base_image_ref | c2d3e37c-b14f-4472-9bc2-d75f26ab900d | 0 | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ 5 rows in set (0.00 sec)

2 1

[opendev] git clone fails: "The TLS connection was non-properly terminated"
by Bernd Bausch 24 Jul '19

24 Jul '19

There seems to be something wrong with the opendev server. Is this a glitch, a daily occurrence (backup?), or a deeper problem? This happens around midnight between July 22 and 23 UTC. $ git clone https://opendev.org/openstack/devstack.git -b stable/stein Cloning into 'devstack'... fatal: unable to access 'https://opendev.org/openstack/devstack.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated. Same result when I try to clone any other random repo. I can get Devstack from Github, but when setting up the cloud, stack.sh wants to clone everything from git.openstack.org, which fails again. I could replace all the hardcoded /git.openstack.org/ strings with the equivalent /github.com/ URL, but I am not sure if I can trust that all Github repos are up to date. Bernd.

4 7

[devstack] Devstack fails with requirement pyparsing===2.4.1
by Bernd Bausch 24 Jul '19

24 Jul '19

Trying to install Devstack Stein on freshly installed Ubuntu 18.04. stack.sh fails with this: Downloading https://files.pythonhosted.org/packages/6f/7c/ebba93a2d3e09fd0220e1adfdb823… (131kB) Collecting pyparsing===2.4.1 (from -c https://releases.openstack.org/constraints/upper/master (line 492)) Could not find a version that satisfies the requirement pyparsing===2.4.1 (from -c https://releases.openstack.org/constraints/upper/master (line 492)) (from versions: 1.4.6, 1.4.7, 1.4.8, 1.4.11, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.7, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.1.7, 2.1.8, 2.1.9, 2.1.10, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0) No matching distribution found for *pyparsing===2.4.1* (from -c https://releases.openstack.org/constraints/upper/master (line 492)) (red highlight is mine) The most recent version of pyparsing <https://pypi.org/project/pyparsing/>is 2.4.0. I would file a bug, but I don't know against which project. Who owns https://releases.openstack.org/constraints/upper/master? Or is there a misunderstanding somewhere? My local.conf is below - it's based on a Rocky local.conf (which does work on Rocky) and might be incorrect. Bernd. [[local|localrc]] ########################################################################### ### Passwords ADMIN_PASSWORD=MYPASSWORD DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD ########################################################################### ### Logging LOG_COLOR=False LOGFILE=$DEST/logs/stack.sh.log ########################################################################### ### Networking IP_VERSION=4 HOST_IP=192.168.1.200 ########################################################################### ### Software sources # use https since the vlabs proxy seems to block the git:// scheme GIT_BASE=https://git.openstack.org # Ensure the latest software is installed PIP_UPGRADE=True ########################################################################### ### Glance DOWNLOAD_DEFAULT_IMAGES=False IMAGE_URLS="http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" # Glance's image import doesn't work when Glance runs as a UWSGI server # For this setting to take effect, devstack/lib/glance must be patched GLANCE_USE_UWSGI=False ########################################################################### ### Swift # by default, Devstack interactively asks for a hash value. Avoid this. SWIFT_HASH=123 enable_service swift ########################################################################### ### Heat enable_plugin heat https://git.openstack.org/openstack/heat stable/stein enable_service h-eng h-api h-api-cfn h-api-cw enable_plugin heat-dashboard https://git.openstack.org/openstack/heat-dashboard stable/stein enable_service heat-dashboard ########################################################################### ### Cinder: Enable backup enable_service c-bak ########################################################################### ### Telemetry: Use Gnocchi as sample repository ### and configure medium archive policy CEILOMETER_BACKEND=gnocchi GNOCCHI_ARCHIVE_POLICY=medium enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer stable/stein enable_plugin aodh https://git.openstack.org/openstack/aodh stable/stein ########################################################################### ### Neutron: Enable internal DNS resolution in ml2_conf.ini Q_ML2_PLUGIN_EXT_DRIVERS=port_security,dns_domain_ports ########################################################################### ### Post config for Neutron (DNS) and Nova (allow KVM in Vmware machine) [[post-config|$NEUTRON_CONF]] [DEFAULT] dns_domain = devstack.org. [[post-config|/etc/nova/nova-cpu.conf]] [libvirt] # set machine type to work around bug # https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1691109 hw_machine_type = x86_64=pc-i440fx-xenial

5 4

[mistral] Office hour in ~5 min
by Renat Akhmerov 24 Jul '19

24 Jul '19

Hi, We’ll have an office hour in about 5 mins. Join us at #openstack-mistral if you have anything to discuss. Thanks Renat Akhmerov @Nokia

1 0

[Kolla][Keystone] No "Domains" found in identity area.
by Eddie Yen 23 Jul '19

23 Jul '19

Hi, I'm using kolla-ansible stable/rocky release to deploy Openstack Try to test with multi-domains requirements. I'm do nothing but add "horizon_keystone_multidomain: True" into globals.yml After deployment, it works because I have to enter the domain name at login screen. Logged in with default domain and admin account. But I cannot found any "Domains" section in identity block. Is there something I miss?

1 2

[keystone] Keystone Train Midcycle Report
by Colleen Murphy 23 Jul '19

23 Jul '19

This week the keystone team conducted a two-day virtual midcycle. Thanks to everyone who participated and contributed to making this a successful and productive meeting! This email will be a short summary of the key takeaways from the last two days. The full agenda and meeting notes can be found on this etherpad: https://etherpad.openstack.org/p/keystone-train-midcycle-topics # Caching Guide We've long been proposing to document best practices for configuring caching for keystone. We began the meeting with a discussion about what exactly should be contained in such a guide, starting with anecdotes from the team about our own operational experience with configuring memcached for keystone. We quickly realized the guide needed at least two parts, one for keystone itself and one for keystonemiddleware. We also made two technical observations, first that the default value for the memcache_socket_timeout option in oslo.cache could be reduced[1] and second that the memcache_socket_timeout option exposed in keystone is not actually used by keystone and needs to be removed[2]. We talked about security strategies for memcached: oslo.cache exposes encryption capabilities which is required for organizations that consider the data stored in memcached to be "at rest", but it also hurts performance. An alternative strategy that could be recommended in the guide is to set up TLS tunneling to secure in-flight data without impeding fetching performance. The initial caching guide update was proposed here[3]. This guide will be a living document and will continue to grow as we gather real-life data from operators. After this initial update, we'll start taking steps to ensure that keystone requires caching be set up in order to operate. [1] https://review.opendev.org/672063 [2] https://bugs.launchpad.net/keystone/+bug/1837407 [3] https://review.opendev.org/672120 # Community Goals We then had a short status update session on the various community goals we need to meet. The mutable config goal from a few cycles ago is not yet complete but has an implementation proposed[4] which needs more eyes and testing. The PDF generation goal implementation is underway[5] but there are some issues already being tracked by the community[6]. We will need to work to ensure the issues we're seeing are noted and addressed. The last of the Python3 Train job changes for keystone repos were merged, so I think we've completed our end of that goal. We also already have a patch proposed to add the ipv6-only job[7] which is just waiting on the base job to merge in tempest. [4] https://review.opendev.org/585417 [5] https://review.opendev.org/669982 [6] https://etherpad.openstack.org/p/pdf-goal-train-common-problems [7] https://review.opendev.org/671903 # Milestone 2 Train Retrospective We closed out the first day with a mid-cycle retrospective[8]. Some key areas for improvement from this were: * Do more video conferencing - The real-time face-to-face communication is really effective for getting to the heart of discussions and making decisions quickly - Since it is looking likely that most of us will not be at the next PTG in Shanghai, we can set up a pre- and post- PTG video meeting to plan the cycle and catch people up after the IRL PTG - Milestone-ly video meetings are working for helping us keep pace and sync up - We can keep a queue of potential topics and make sure we have a meeting when there are enough topics to discuss - We'll try using video conferencing for office hours - We'll try Jitsi again even though there were some technical issues last time since it is OSS and supports recording * Improve office hours - Office hours still don't feel useful and haven't happened regularly, but we still think they have potential - We'll use the weekly newsletter to commit to the topic the week before, which gives the team a better opportunity to prepare for it and can also attract interest from people outside the keystone team - Use video conferencing as mentioned above * Improve the weekly newsletter - We'll use the weekly newsletter to call out action items from the last meeting - In addition to providing statistics about the number of open reviews, we'll highlight certain reviews that need attention based on various criteria such as age, priority with regard to cycle commitments, community goals, and specific review requests. - We'll announce the following week's office hours topic in the newsletter [8] https://etherpad.openstack.org/p/keystone-train-midcycle-retrospective # System Scope and Default Roles The second day of the midcycle was used to go through our list of open system scope[9] and default roles[10] bugs and work through what scopes and roles each API should support. The notes in the etherpad largely capture every decision fairly well, but a few issues worth highlighting are: * Implied roles[11]: the regular roles API is currently system-scope only, even though it's possible to have domain-specific roles, because for the most part for a role to be useful the role's creator must also have the ability to configure policy, which domain admins do not. Implied roles still require access to policy configuration except in the case where you are only using implied roles for aliasing, in which case it may make sense for a domain admin to have the ability to create an implied role relationship. We decided to start with making the implied roles API system-scope only for now and to open a separate bug to discuss how to deal with domain-specific roles and implied roles. * Revocation list[12]: we're currently not enforcing policy on this API at all because it can only ever return a 410 Gone or a 403 Forbidden, never a success, so we will just deprecate the policy entirely rather than updating it to work with system scope or default roles. We reevaluated the priority on some of the bugs in the list. However, we noted that while some APIs are more heavily used than others, we can't change enforce_scope to True until all of them are completed - picking and choosing the most important ones and letting the others slide is not really an option. [9] https://bugs.launchpad.net/keystone/+bugs?field.tag=system-scope [10] https://bugs.launchpad.net/keystone/+bugs?field.tag=default-roles [11] https://bugs.launchpad.net/keystone/+bug/1805371 [12] https://bugs.launchpad.net/keystone/+bug/1818845 If I've left out any important details, don't hesitate to add them to this thread. Thanks again to everyone who was able to join the meeting! Thanks also to Julia Kreger for her advice on how to hold one of these virtual midcycles, it proved to be very useful and this first-timer really appreciated it :) Colleen

1 0

[tripleo] CI scenario001-standalone broken
by John Fulton 23 Jul '19

23 Jul '19

As per $subject [0]. scenario010 is affected too. This correlates with the promotion of the ceph-ansible 4rc11 RPM [1]. 001 jobs using rc9 do not have this problem. I'm looking into a fix, e.g. removing the storage7-ceph-nautilus-release tag from the rc11 package as TripleO CI uses the version of ceph-ansible which is tagged for release [2]. I'll keep this thread updated with the status and when it's fixed. John [0] http://logs.openstack.org/74/670574/4/check/tripleo-ci-centos-7-scenario001… [1] https://cbs.centos.org/koji/buildinfo?buildID=26343 [2] https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/r…

3 4

[Diversity] Grace Hopper Lead Co-Mentor(s) needed
by Amy Marrich 23 Jul '19

23 Jul '19

OpenStack has participated in Open Source Day at the Grace Hopper Conference for the last 5 years and this year it is in Orlando on October 3rd. Due to unforeseen circumstances we are one lead mentor short to participate this year and are looking for someone willing to help lead the session. The Anita Borg Institute will provide a ticket for the entire conference, the downside is you or your company will be responsible for your travel, lodging and food. This year CityNetwork is supplying the VM infrastructure for us to use. The session submitted is as follows: Participants will sign up for the necessary accounts needed to contribute to OpenStack and install Git and Gerrit. They will configure their systems and learn how to file bugs and review code within the Open Dev infrastructure. Due to time and resource requirements the groups will be given a virtual machine running Devstack in which to create their own theme for the Horizon Dashboard. The group will decide on which humanitarian effort they wish to support but suggestions will be provided for those groups that can not decide. The group will also work as a team to locate the necessary documentation on the OpenStack site if they have not done so in preparation. The selected Project Manager will lead these discussions as well as making sure efforts are staying on track to meet the deadline of the demonstrations. The Developers in the group will work to locate the necessary files and directories on the VM in order to complete their task. They will be in charge of creating/changing the necessary files to create a new theme utilizing SCSS and restarting any services needed to implement. The Graphic Designer if present will create any necessary graphics for the theme and will either scp them themselves to the virtual machine or provide to the developer. If there is no designated graphic designer the PM and devs will divide this duty up. After the themes are designed relevant files will be scped back down to the participants machines and they will be able to commit their code for review in a sandbox environment for other groups to review. If you are interested in participating please let me know by July 20th as we need to send additional information to the OSD folks on July 21st. Any questions please reach out on the lists, IRC, or privately! Thanks, Amy (spotz)

1 1

[Octavia]-Seeking performance numbers on Octavia
by Singh, Prabhjit 23 Jul '19

23 Jul '19

Hi I have been trying to test Octavia with some traffic generators and my tests are inconclusive. Appreciate your inputs on the following * It would be really nice to have some performance numbers that you guys have been able to achieve for this to be termed as carrier grade. * Would also appreciate if you could share any inputs on performance tuning Octavia * Any recommended flavor sizes for spinning up Amphorae, the default size of 1 core, 2 Gb disk and 1 Gig RAM does not seem enough. * Also I noticed when the Amphorae are spun up, at one time only one master is talking to the backend servers and has one IP that its using, it has to run out of ports after 64000 TCP concurrent sessions, id there a way to add more IPs or is this the limitation * If I needed some help with Octavia and some guidance around performance tuning can someone from the community help Thanks & Regards Prabhjit Singh

3 5

[ironic] Using ironic in a devstack environment to deploy OS on bare-metal host.
by Guannan GN2 Sun 23 Jul '19

23 Jul '19

Hi team, I met some problem when using ironic in a devstack environment to deploy OS on bare-metal host. By now I can successfully boot into the tiny os and set ip, also I can ping the ip from my devstack server. However, when the tiny os running some configuration, it will failed after about 5 minutes. I found some log information in "/var/log/ironic-python-agent.log" in the tiny os that may relate to the problem: "DEBUG oslo_concurrency.processutils [-] u'iscsistart -f' failed. Not Retrying. execute /usr/local/lib/python2.7/site-packages/oslo_concurrency/processutils.py:457" "DEBUG root [-] No iscsi connection detected. Skipping iscsi. Error: [Errno 2] No such file or directory _check_for_iscsi /usr/local/lib/python2.7/site-packages/ironic_python_agent/hardware.py:107" "WARNING root [-] The root device was not detected in 27 seconds: DeviceNotFound: Error finding the disk or partition device to deploy the image onto: No suitable device was found for deployment - root device hints were not provided and all found block devices are smaller than 4294967296B." "WARNING root [-] Can't find field vendor for device usb0 in device class net: IOError: [Errno 2] No such file or directory: '/sys/class/net/usb0/device/vendor'" "WARNING root [-] Can't find field vendor for device tunl0 in device class net: IOError: [Errno 2] No such file or directory: '/sys/class/net/tunl0/device/vendor'" "DEBUG root [-] No Mellanox devices found evaluate_hardware_support /usr/local/lib/python2.7/site-packages/ironic_python_agent/hardware_managers/mlnx.py:84" And when it deploy faild and I run command "openstack baremetal node show server-51", it will show following information on 'last_error' field: "Asynchronous exception: Node failed to deploy. Exception: Connection to agent failed: Failed to connect to the agent running on node f8794658-bc69-40cb-9673-be75f78ced21 for invoking command iscsi.start_iscsi_target. Error: ('Connection aborted.', BadStatusLine("''",)) for node" I'm trying to debug, but have not find the root cause, please give me your advice if someone have experience on it. Thank you! Best Regards, Guannan

2 1

[dev][magnum] Magnum event notifications not having relevant content related to clusters
by Emnauel Andrecut 23 Jul '19

23 Jul '19

I'm writing about Magnum event notifications that do not contain any relevant information about the related Cluster (missing cluster id, other attributes..) As I've seen the PyCADF library receives no targetId and so the id (and mostly any other id) is randomly generated. Do you plan to add more information about the cluster on create success/pending events. Or would you accept a patch that does that?

3 2

[all][meta-sig][containers-sig] Creation of Containers SIG
by Jean-Philippe Evrard 23 Jul '19

23 Jul '19

Hello everyone, I have discussed with some of you about the fact OpenStack is lacking a a group interested in sharing the practices for the building/management of OpenStack application containers. Every container related project is doing its own way (and it's probably fine), but very few experiences are shared (and it's sad). I started an etherpad [1] to see areas where container projects/people could collaborate. I already received significant feedback on how to improve some of my practices, so I think it's already a positive experience for me to have this group started. Anyway, long story short, I believe we should create a SIG for this, so I proposed the addition of the containers-sig into governance [2]. We have enough work to started this group nowadays (with documentation, bindep shared practices, and improvement of projects). I suppose technologies will evolve over time, so I expect a SIG is the best tool to do this job of *for example* gathering experience, as it's not really time bound, and it _is_ a quite special interest. If you're interested by learning what is going on in the container building world for OpenStack, please add [containers-sig] to your openstack-discuss topics watch list. Regards, Jean-Philippe Evrard (evrardjp) [1]: https://etherpad.openstack.org/p/containers-sig [2]: https://review.opendev.org/#/c/666001/

1 0

[kolla] Stein release now available
by Mark Goddard 23 Jul '19

23 Jul '19

Hi, I'm pleased to announce that the first official Stein cycle releases for Kolla and Kolla Ansible are now available - both 8.0.0. A lot of work has gone into this release, including improving the CI tests, making it the most well tested release yet. Thanks to everyone who contributed. Release notes: https://docs.openstack.org/releasenotes/kolla/stein.html https://docs.openstack.org/releasenotes/kolla-ansible/stein.html Join us on #openstack-kolla to help make the Train cycle release even better. Cheers, Mark

1 0

[telemetry] Looking for new meeting time
by Trinh Nguyen 22 Jul '19

22 Jul '19

Hi team, I've changed job and could not afford doing the meeting at 02:00 UTC on Thursday anymore. My possible time slot is from 13:00-15:00 UTC Tuesday-Thursday. Please propose a better time frame for our team meeting. Sincerely, -- *Trinh Nguyen* *www.edlab.xyz <https://www.edlab.xyz>*

2 3

[nova] FYI to out-of-tree drivers: adding a positional argument to finish_revert_migration()
by Artom Lifshitz 22 Jul '19

22 Jul '19

[1] will most likely merge in its current shape or something very close to it. It adds a `migration` positional argument to the finish_revert_migration() virt driver method. `migration` is the, err, migration that's being reverted. If you have out of tree virt drivers, please update them to match this new method signature. Thanks! [1] https://review.opendev.org/#/c/668631/

1 0

[tripleo] TripleO CI Summary: Sprint 33
by Rafael Folco 22 Jul '19

22 Jul '19

Greetings, The TripleO CI team has just completed Sprint 33 / Unified Sprint 12 (Jun 27 thru Jul 17). The following is a summary of completed work during this sprint cycle: - Tested check jobs for periodic image/container builds on RHEL8 and resolved its dependencies on rhui repositories. - Enabled tripleo-repos to support RHEL8. - Addressed changes in kolla to support RHEL8. - Created a nodepool image on RDO for RHEL8 w/ elements and dependencies installed. - Improved container build script to consolidate kolla build logs and ease troubleshooting. - Manually pushed rhel containers to RDO registry and identified gaps in authentication workflow for the container build jobs using buildah tooling. - Modified scripts and post playbooks to upload RHEL8 overcloud images. - Created RHEL8 jobs to build a periodic pipeline in the RDO Software Factory and provide feedback for CentOS8 coverage. - Resumed scenario007 and featureset039 job updates upstream. - Promotion status: green on all branches at most of the sprint. The planned work for the next sprint [1] are: - Get RDO on RHEL8 build jobs working in periodic pipeline and bootstrap standalone and featureset001 jobs to consume RHEL8 images/containers. - Get featureset039 job to run novajoin tests and merge scenario007 job changes. - Review templates for branched and non-branched projects to have more control on what jobs are being run and reduce queue lengths. - Resume the design work for a staging environment to test changes in the promoter server for the multi-arch builds. The Ruck and Rover for this sprint are Marios Andreou (marios) and Rafael Folco (rfolco). Please direct questions or queries to them regarding CI status or issues in #tripleo, ideally to whomever has the ‘|ruck’ suffix on their nick. Ruck/rover notes are being tracked in etherpad [2]. Thanks, rfolco [1] https://tree.taiga.io/project/tripleo-ci-board/taskboard/unified-sprint-13 [2] https://etherpad.openstack.org/p/ruckroversprint13

1 0

[neutron] Bug Deputy July 15 - 21
by Hongbin Lu 22 Jul '19

22 Jul '19

Hi all, Below is the list of bugs last week: * https://bugs.launchpad.net/neutron/+bug/1837252 IFLA_BR_AGEING_TIME of 0 causes flooding across bridges (Incompleted) * https://bugs.launchpad.net/neutron/+bug/1837169 Stein: testing _make_floatingip fails in Debian Sid (Invalid) * https://bugs.launchpad.net/neutron/+bug/1837012 Neutron-dynamic-routing tempest tests can't be run all in one job (Medium) * https://bugs.launchpad.net/neutron/+bug/1836834 [RFE] introduce distributed locks to ipam * https://bugs.launchpad.net/neutron/+bug/1836642 Metadata responses are very slow sometimes (High) * https://bugs.launchpad.net/neutron/+bug/1836595 test_server_connectivity_cold_migration_revert failing (Medium) * https://bugs.launchpad.net/neutron/+bug/1836565 Functional test test_keepalived_state_change_notification may fail do to race condition (Medium) Best regards, Hongbin

1 0

[tc][ptl][infra] Resolution: Mandatory Repository Retirement
by Jeremy Stanley 22 Jul '19

22 Jul '19

When an effort ceases to be an official part (deliverable) of OpenStack with the intent to continue development outside OpenStack's technical governance, the maintainers of that codebase may end up later abandoning it or making other choices at odds with OpenStack policies with no clear indication to consumers. Some folks may have been using it when it was still part of OpenStack, and so may continue to have expectations which are no longer in sync with reality. To solve this, a resolution has been proposed asserting that software which is removed from OpenStack's technical governance in the future can only be continued as a fork, so that its original Git repository in the "openstack" namespace on OpenDev can be retired with a clear message letting consumers of that source code know what has happened and where it has gone. Some breathing room is allowed, as the initial draft includes measures to allow any deliverable repositories which move out of OpenStack prior to the end of the current development cycle (Train) to be renamed into new OpenDev namespaces with a redirect before the new policy goes into effect. The text of the proposed resolution can be found here: https://review.opendev.org/670741 Please follow up on the above review with any comments/concerns, or you can express them as a reply to this ML thread, or in private to me or any other TC members. Thanks for reading! -- Jeremy Stanley

1 1

[tc] weekly update
by Mohammed Naser 22 Jul '19

22 Jul '19

Hi everyone, This is the weekly update for what happened inside the Openstack TC. You can get more information by checking for changes in the openstack/governance repository. # Retired project: - TripleO-UI: https://review.opendev.org/#/c/665264/ # General changes: - the 'U' release name poll was changed to China to include all regions, provinces, and cities: https://review.opendev.org/#/c/668071/ - An empty deliverables list was added to refstack (retired team): https://review.opendev.org/#/c/671141/ Thanks! Mohammed

1 0

[openstack-ansible][tripleo] creating collaborative acl
by Mohammed Naser 22 Jul '19

22 Jul '19

Hi everyone, I briefly brought this up in the TripleO Ansible meeting however there seems to be more work happening together on OpenStack Ansible roles from the TripleO side (alongside cross-project gating, which is awesome!) Having said that, it seems like we're building more of an upstream relationship to TripleO but I think there is a lot of benefit of making it more of a collaboration between the teams. Therefore, I'd like to propose that we create a Gerrit group called "openstack-ansible-collab". This ACL will be seeded with tripleo-ansible-core and openstack-ansible-core and it will ideally be placed on two initial repositories: - openstack/ansible-config_template - openstack/ansible-role-python_venv_build - openstack/openstack-ansible-os_tempest Those are repositories that are actively being used by TripleO and they are even getting CI tested from TripleO. They make the most sense to start with. TripleO has been around for a while and they have a solid core team that I (personally) feel okay with having core to those repos for now (and more as we work on them together). In terms of governance, these projects will remain under OpenStack-Ansible governance. Please let me know what you think of this proposal? Thanks, Mohammed

6 5

Cinder data path and control path
by Jyoti Dahiwele 22 Jul '19

22 Jul '19

Dear Team, Plz clear doubt about cinder control path and data path. If control path goes off and data path continue then still vm continues to use attached volumes.

2 1

[keystone] Returning core Guang Yee
by Colleen Murphy 21 Jul '19

21 Jul '19

Hi everyone, I'm thrilled to announce that Guang Yee has agreed to rejoin the keystone core reviewer team. Guang is a former core who was instrumental in many of keystone's key features. We're lucky to have him back to lend his experience and knowledge to our current challenges. Welcome back, Guang! Colleen

6 5

[kolla][ironic] Paternity leave
by Mark Goddard 20 Jul '19

20 Jul '19

Hi, I will be on paternity leave for the next few weeks. Please speak to another core in #openstack-kolla for Kolla or Kayobe issues. I'll check in from time to time. Cheers, Mark PS boy, Frank, very cute.

3 2

[keystone] Shanghai PTG planning
by Colleen Murphy 20 Jul '19

20 Jul '19

Hi team, The foundation has asked us to let them know whether we'll be needing PTG space in Shanghai. The keystone team usually has good attendance at PTGs and makes good use of the time, but I know this next one may be hard for some people to attend so I don't want to automatically assume we'll use it this time. If you wish to attend the next PTG and have a semi-reasonable amount of confidence that you may be able to, please add your name to the etherpad: https://etherpad.openstack.org/p/keystone-shanghai-ptg Please do this by Monday, August 5 so that I have time to respond to the foundation by August 11. Feel free to also start brainstorming topics for the PTG (no deadline on this). Colleen

2 1

Unable to ssh openstack instance
by Manula Thantriwatte 20 Jul '19

20 Jul '19

Hi All, I'm trying to ssh openstack image I have created using cirros image. But I'm getting following error. ssh: connect to host 172.24.4.147 port 22: No route to host But when I go to the instance console I'm able to login. For the security groups I have enable port 22. I have associated floating IP as well. How to resolve this? Thanks ! -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika(a)gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika <http://lk.linkedin.com/in/manulachathurika>* blog : http://manulachathurika.blogspot.com/

5 11

local storage + remote replication
by Volodymyr Litovka 20 Jul '19

20 Jul '19

Dear colleagues, this question isn't about Openstack, it's rather general question about cloud data storing. It is obvious, that distributed network storage is generally slower than local storage, while is much more reliable. Keeping in mind, that reliability of hardware components became better and better with time (e.g. MTBF of HP DL380G2 is almost 14 years), power systems are reliable enough (if built with reliability in mind), it is safe to say that the most possible source of problems is failure of storage itself, not motherboard or NIC or power supply. Why not to use kind of caching, quickly saving data on local storage, ACKing the operation and replicating changed blocks to remote storage in background during idle time? Having a local RAID1 (with checksums to allow distinguish between failed and correct copy of the data block) + remote replication will give a high speed and enough level of reliability. The only case of temporary slowing down is migration to another node (not pretty often event?) with gradual back replication (from remote storage to local, as the VM will request data) which will last some time from start at new node. Are there serious concerns against such approach? I will appreciate your opinion on this topic. Thank you! -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison

1 0

[keystone] Keystone Team Update - Week of 15 July 2019
by Colleen Murphy 19 Jul '19

19 Jul '19

# Keystone Team Update - Week of 15 July 2019 ## News ### Returning core Welcome back, Guang![1] [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007918.html ### Midcycle next week The virtual midcycle will take place next week on Monday and Tuesday[2]. We will (try to) use Gotomeeting for the call, the details for which are in the etherpad. [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007593.html ### Train specs merged All specs proposed for the Train cycle are now merged! ## Open Specs Train specs: https://bit.ly/2uZ2tRl Ongoing specs: https://bit.ly/2OyDLTh ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 8 changes this week. ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 40 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ## Bugs This week we opened 5 new bugs and closed 4. Bugs opened (5) Bug #1837061 (keystone:Wishlist) opened by Morgan Fainberg https://bugs.launchpad.net/keystone/+bug/1837061 Bug #1836568 (keystone:Undecided) opened by Attila Fazekas https://bugs.launchpad.net/keystone/+bug/1836568 Bug #1836650 (keystone:Undecided) opened by Rafael Weingartner https://bugs.launchpad.net/keystone/+bug/1836650 Bug #1836872 (keystone:Undecided) opened by Yang Youseok https://bugs.launchpad.net/keystone/+bug/1836872 Bug #1837010 (keystone:Undecided) opened by Yang Youseok https://bugs.launchpad.net/keystone/+bug/1837010 Bugs closed (3) Bug #1832267 (keystone:Undecided) https://bugs.launchpad.net/keystone/+bug/1832267 Bug #1836650 (keystone:Undecided) https://bugs.launchpad.net/keystone/+bug/1836650 Bug #1836872 (keystone:Undecided) https://bugs.launchpad.net/keystone/+bug/1836872 Bugs fixed (1) Bug #1813057 (keystone:Medium) fixed by Guang Yee https://bugs.launchpad.net/keystone/+bug/1813057 ## Milestone Outlook https://releases.openstack.org/train/schedule.html Next week is spec freeze, which we've already met. One month from now is feature proposal freeze, by which point all code for new features should be proposed and ready for review - no WIPs. This gives us four weeks to review and incorporate feedback before feature freeze. ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter

1 0

[tripleo] specs for removing Puppet from TripleO
by Emilien Macchi 19 Jul '19

19 Jul '19

As part of the Simplification of TripleO effort [1], I pushed a specification proposal to remove Puppet from TripleO. This is a *huge* effort and to be clear I'm still unsure it is really doable; but I would like our team to explore this path and see if there is enough value out of it. https://review.opendev.org/#/c/671563 HTML output with latest patchset: http://logs.openstack.org/63/671563/4/check/openstack-tox-docs/c511ace/html… Feel free to give feedback directly in the specs or here in this thread. If we agree to move forward, my hope is we can push for a first "tech-preview" interface in Train; but keep in mind this would be a multi-cycles effort. [1] https://docs.google.com/document/d/1vY9rsccgp7NHFXpLtCFTHQm14F15Tij7lhn5X_P… -- Emilien Macchi

1 0

How to restart DevStack
by Manula Thantriwatte 19 Jul '19

19 Jul '19

Hi All, When I shutdown my PC, how to restart devstack. Is it ok to run ./stack.sh again or do I need to do it from the beginning? Thanks ! -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika(a)gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika <http://lk.linkedin.com/in/manulachathurika>* blog : http://manulachathurika.blogspot.com/

6 5

vm guest creation
by Mauricio Tavares 19 Jul '19

19 Jul '19

If I am creating a kvm guest, I take it is doing something like virt-install \ --name desktop1 \ --disk path=./moocow.qcow2 \ --ram 4098 --vcpus 2 \ --os-type linux \ [....] --arch x86_64 Where does it do that? And, how much control per guest do I have? I am looking at https://docs.openstack.org/mitaka/config-reference/compute/hypervisor-kvm.h…. What I see is that I can only define across the border options in nova.conf. Also, are there any docs on configuring the libvirt besides https://wiki.openstack.org/wiki/LibvirtXMLConfigAPIs?

1 0

[goal][python3] Train unit tests weekly update (goal-8)
by Corey Bryant 19 Jul '19

19 Jul '19

This is the goal-8 weekly update for the "Update Python 3 test runtimes for Train" goal [1]. There are 8 weeks remaining for completion of Train community goals [2]. == What's the Goal? == To ensure (in the Train cycle) that all official OpenStack repositories with Python 3 unit tests are exclusively using the 'openstack-python3-train-jobs' Zuul template or one of its variants (e.g. 'openstack-python3-train-jobs-neutron') to run unit tests, and that tests are passing. This will ensure that all official projects are running py36 and py37 unit tests in Train. For complete details please see [1]. == Ongoing Work == Patches have been submitted for all applicable projects except for one, OpenStack Charms. The charms have a mid-cycle freeze this week. I hope to have those submitted next week. Open patches needing reviews: https://review.openstack.org/#/q/topic:python3-train+is:open Failing patches: https://review.openstack.org/#/q/topic:python3-train+status:open+(+label:Ve… Patch automation scripts needing review: https://review.opendev.org/#/c/666934 == Completed Work == Merged patches: https://review.openstack.org/#/q/topic:python3-train+is:merged == How can you help? == Please take a look at the failing patches and help fix any failing unit tests for your projects. Python 3.7 unit tests will be self-testing in Zuul. == Reference Material == [1] Goal description: https://governance.openstack.org/tc/goals /train/python3-updates.html [2] Train release schedule: https://releases.openstack.org/train/schedule.html (see R-5 for "Train Community Goals Completed") Storyboard: https://storyboard.openstack.org/#!/story/2005924 Porting to Python 3.7: https://docs.python.org/3/whatsnew/3.7.html#porting-to-python-3-7 Python Update Process: https://opendev.org/openstack/governance/src/branch/master/resolutions/2018… Train runtimes: https://opendev.org/openstack/governance/src/branch/master/reference/runtim… Thanks, Corey

1 0

Instructions for spooling up an instance
by Mauricio Tavares 19 Jul '19

19 Jul '19

I understand you create a flavor and an image (which can be customized with the properties from https://docs.openstack.org/glance/latest/admin/useful-image-properties.html) which can then be fed to openstack server create thingie. But, where is the script (think something like kickstart to see what I have in mind; if I am off, I am off) that defines how the hard drive will be partitioned and which interfaces (if more than one) will be turned on and which will be left untouched. You know, housekeeping things that are done before you feed the "--user-data" stuff. Where is that defined? Can I customize it?

4 4

[queens][cinder] emc fc driver
by Ignazio Cassano 19 Jul '19

19 Jul '19

Hello, Does anyone use cinder with emc san via FC ? Does it work ? Regards Ignazio

2 2

Octavia: Could not retrieve certificate when create HTTPS listener using application credentials
by Pawel Konczalski 19 Jul '19

19 Jul '19

Hi, i try to create a Octavia HTTPS listener by using application credentials but get this error: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58…', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…'] (HTTP 400) (Request-ID: req-088d6eb0-a285-4089-bc11-ff0c3097123e) # openstack secret list +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration | +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58… | cert2 | 2019-07-19T13:42:21+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | opaque | cbc | None | | https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7… | cert1 | 2019-07-19T13:42:12+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | opaque | cbc | None | +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ # openstack loadbalancer listener create foo-lb1 \ --name foo-lb1-https-listener \ --protocol-port 443 \ --protocol TERMINATED_HTTPS \ --insert-headers X-Forwarded-For=true,X-Forwarded-Proto=true \ --default-tls-container=https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09 \ --sni-container-refs https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7… https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58… -------------------------------- Starting new HTTPS connection (1): octavia.service.dev.example.com:443 https://octavia.service.dev.example.com:443 "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 779 RESP: [200] Connection: keep-alive Content-Length: 779 Content-Type: application/json Date: Fri, 19 Jul 2019 13:56:24 GMT Server: WSGIServer/0.1 Python/2.7.15rc1 x-openstack-request-id: req-50b5a3bb-21ec-4a46-8d5c-61035afd3423 RESP BODY: {"loadbalancers": [{"provider": "amphora", "description": "", "admin_state_up": true, "pools": [{"id": "169722d1-0a73-4283-bb42-aee5b662e2e2"}], "created_at": "2019-07-19T13:34:52", "provisioning_status": "ACTIVE", "updated_at": "2019-07-19T13:39:34", "vip_qos_policy_id": null, "vip_network_id": "2064c61c-64a1-466f-983a-af435ae1d51c", "listeners": [{"id": "169a91f9-ef5c-4d38-8449-e24b64cf082d"}], "tenant_id": "9646533a8d834978a868e81c9b9a39cf", "vip_port_id": "dcfc6e44-4092-4f2b-bd50-24e02abb078f", "flavor_id": "", "vip_address": "10.0.1.4", "vip_subnet_id": "787035dc-add4-4227-844a-1cf803625abc", "project_id": "9646533a8d834978a868e81c9b9a39cf", "id": "e2ed48ab-3261-422f-b9b5-a5aa63486ae7", "operating_status": "OFFLINE", "name": "foo-lb1"}], "loadbalancers_links": []} GET call to https://octavia.service.dev.example.com/v2.0/lbaas/loadbalancers used request id req-50b5a3bb-21ec-4a46-8d5c-61035afd3423 REQ: curl -g -i -X POST https://octavia.service.dev.example.com/v2.0/lbaas/listeners -H "Content-Type: application/json" -H "User-Agent: openstacksdk/0.19.0 keystoneauth1/3.11.1 python-requests/2.20.1 CPython/2.7.15+" -H "X-Auth-Token: {SHA256}6414e14f4e78940902b11c89567689e3cc0d3ea62227b87a1e19361685c83584" -d '{"listener": {"insert_headers": {"X-Forwarded-For": "true", "X-Forwarded-Proto": "true"}, "protocol": "TERMINATED_HTTPS", "name": "foo-lb1-https-listener", "default_tls_container_ref": "https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…", "sni_container_refs": ["https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…", "https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58…"], "admin_state_up": true, "protocol_port": 443, "loadbalancer_id": "e2ed48ab-3261-422f-b9b5-a5aa63486ae7"}}' https://octavia.service.dev.example.com:443 "POST /v2.0/lbaas/listeners HTTP/1.1" 400 357 RESP: [400] Connection: keep-alive Content-Length: 357 Content-Type: application/json Date: Fri, 19 Jul 2019 13:56:27 GMT Server: WSGIServer/0.1 Python/2.7.15rc1 x-openstack-request-id: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca RESP BODY: {"debuginfo": null, "faultcode": "Client", "faultstring": "Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58…', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…']"} POST call to https://octavia.service.dev.example.com/v2.0/lbaas/listeners used request id req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca Request returned failure status: 400 Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58…', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand result = cmd.run(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run column_names, data = self.take_action(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/osc/v2/listener.py", line 168, in take_action json=body) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper request_id=e.request_id) OctaviaClientException: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58…', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) clean_up CreateListener: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58…', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) Traceback (most recent call last): File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/shell.py", line 136, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 279, in run result = self.run_subcommand(remainder) File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/shell.py", line 176, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand result = cmd.run(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run column_names, data = self.take_action(parsed_args) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/osc/v2/listener.py", line 168, in take_action json=body) File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper request_id=e.request_id) OctaviaClientException: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58…', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7…'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) ------------------------------ This issue occurs only when application credentials are used. Creation of HTTP listener with applications credentials works fine, also creation of HTTPS listener when user are authenticated by user / password. Does somebody know which additional ACLs / permissions are required to fix this? BR Pawel

2 1

[Manila] CephFS deferred deletion
by Jose Castro Leon 19 Jul '19

19 Jul '19

Dear all, Lately, one of our clients stored 300k files in a manila cephfs share. Then he deleted the share in Manila. This event make the driver unresponsive for several hours until all the data was removed in the cluster. We had a quick look at the code in manila [1] and the deletion is done first by calling the following api calls in the ceph bindings (delete_volume[1] and then purge_volume[2]). The first call moves the directory to a volumes_deleted directory. The second call does a deletion in depth of all the contents of that directory. The last operation is the one that trigger the issue. We had a similar issue in the past in Cinder. There, Arne proposed to do a deferred deletion of volumes. I think we could do the same in Manila for the cephfs driver. The idea is to continue to call to the delete_volume. And then inside a periodic task in the driver, asynchronously it will get the contents of that directory and trigger the purge command. I can propose the change and contribute with the code, but before going to deep I would like to know if there is a reason of having a singleton for the volume_client connection. If I compare with cinder code the connection is established and closed in each operation with the backend. If you are not the maintainer, could you please point me to he/she? I can post it in the mailing list if you prefer Cheers Jose Castro Leon CERN Cloud Infrastructure [1] https://github.com/openstack/manila/blob/master/manila/share/drivers/cephfs… [2] https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L… [2] https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L… PS: The issue was triggered by one of our clients in kubernetes using the Manila CSI driver

3 4

Linstor volume driver
by Jyoti Dahiwele 19 Jul '19

19 Jul '19

Dear Team, Does stein cinder support drbd linstor driver?

3 2