May 2020 - openstack-discuss - lists.openstack.org

[Horizon] Failing to configure the Instance Launch dialog
by Bernd Bausch 26 May '20

26 May '20

On an Ussuri Devstack, I would like to launch instances with ephemeral storage from Horizon and remove the volume storage switch from the instance launch dialog. Prior to Train, the dialog was configured with the LAUNCH_INSTANCE_DEFAULTS dictionary in /local_settings.py/. This is my dictionary; I set /create_volume /to /False /and /hide_create_volume /to /True/: LAUNCH_INSTANCE_DEFAULTS = { 'config_drive': False, 'create_volume': False, 'hide_create_volume': True, 'disable_image': False, 'disable_instance_snapshot': False, 'disable_volume': False, 'disable_volume_snapshot': False, 'enable_scheduler_hints': True, } Starting with Train, the dictionary is in /.../openstack_dashboard/defaults.py/, but the changes I made there have no effect. Adding the dictionary to /local_settings.py/ has no effect either. To test whether /defaults.py/ is used at all, I added a syntactically incorrect line, and indeed, the syntax error is logged in /horizon_error.log/. I wonder what I am doing wrong. I am convinced it worked like this on Stein. Any ideas? Bernd Bausch

2 3

[TripleO] External network on compute node
by Reza Bakhshayeshi 26 May '20

26 May '20

Hi all. The default way of compute node for accessing Internet if through undercloud. I'm going to assign an IP from External network to each compute node with default route. But the deployment can't assign an IP to br-ex and fails with: " raise AddrFormatError('invalid IPNetwork %s' % addr)", "netaddr.core.AddrFormatError: invalid IPNetwork ", Actually 'ip_netmask': '' is empty during deployment for compute nodes. I've added external network to compute node role: External: subnet: external_subnet and for network interface: - type: ovs_bridge name: bridge_name mtu: get_param: ExternalMtu dns_servers: get_param: DnsServers use_dhcp: false addresses: - ip_netmask: get_param: ExternalIpSubnet routes: list_concat_unique: - get_param: ExternalInterfaceRoutes - - default: true next_hop: get_param: ExternalInterfaceDefaultRoute members: - type: interface name: nic3 mtu: get_param: ExternalMtu use_dhcp: false primary: true Any suggestion would be grateful. Regards, Reza

2 6

[Nova][Scheduler] Reducing race-conditions and re-scheduling during creation of multiple high-ressources instances or instances with anti-affinity.
by Laurent Dumont 25 May '20

25 May '20

Hey everyone, We are seeing a pretty consistent issue with Nova/Scheduler where some instances creation are hitting the "max_attempts" limits of the scheduler. Env : Red Hat Queens Computes : All the same hardware and specs (even weight throughout) Nova : Three nova-schedulers This can be due to two different factors (from what we've seen) : - Anti-affinity rules are getting triggered during the creation (two claims are done within a few milliseconds on the same compute) which counts as a retry (we've seen this when spawning 40+ VMs in a single server group with maybe 50-55 computes - or even less 14 instances on 20ish computes). - We've seen another case where MEMORY_MB becomes an issue (we are spinning new instances in the same host-aggregate where VMs are already running. Only one VM can run per compute but there are no anti-affinity groups to force that between the two deployments. The ressource requirements prevent anything else from getting spun on those). - The logs look like the following : - Unable to submit allocation for instance 659ef90e-33b8-42a9-9c8e-fac87278240d (409 {"errors": [{"status": 409, "request_id": "req-429c2734-2f2d-4d2d-82d1-fa4ebe12c991", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on resource provider '35b78f3b-8e59-4f2f-8cad-eaf116b7c1c7'. The requested amount would exceed the capacity. ", "title": "Conflict"}]}) / Setting instance to ERROR state.: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f6d06cca-e9b5-4199-8220-e3ff2e5c2a41. - I do believe we are hitting this issue as well : https://bugs.launchpad.net/nova/+bug/1837955 - In all the cases where the Stacks creation failed, one instance was left in the Build state for 120 minutes and then finally failed. >From what we can gather, there are a couple of parameters that be be tweaked. 1. host_subset_size (Return X number of host instead of 1?) 2. randomize_allocation_candidates (Not 100% on this one) 3. shuffle_best_same_weighed_hosts (Return a random of X number of computes if they are all equal (instance of the same list for all scheduling requests)) 4. max_attempts (how many times the Scheduler will try to fit the instance somewhere) We've already raised "max_attempts" to 5 from the default of 3 and will raise it further. That said, what are the recommendations for the rest of the settings? We are not exactly concerned with stacking vs spreading (but that's always nice) of the instances but rather making sure deployments fail because of real reasons and not just because Nova/Scheduler keeps stepping on it's own toes. Thanks!

3 6

[all][interop] Please suggest capabilities to deprecate-remove and add to the OpenStack Powered Programs
by prakash RAMCHANDRAN 25 May '20

25 May '20

Hi all, Please help the Interop WG to review and finalize the draft version for the next interop guidelines. https://review.opendev.org/gitweb?p=openstack/interop.git;a=blob;f=2020.06.… Link for the DNS add-on: https://review.opendev.org/gitweb?p=openstack/interop.git;a=blob;f=add-ons/… Link for the Heat add-on: https://review.opendev.org/gitweb?p=openstack/interop.git;a=blob;f=add-ons/… This guidelines is intended to cover "Stein" , "Train", "Ussuri" and "Victoria" releases of OpenStack. We request that PTLs or Core team members please confirm any updates you may want to include or exclude in link above. The Projects currently covered under OpenStack Powered Programs include "Keystone", "Glance", "Nova", "Neutron", "Cinder", "Swift" and add-on programs "Designate", "Heat". A more human-readable view of the capabilities can be found in RefStack: https://refstack.openstack.org/#/guidelines We would like have feedback by next Interop call on Friday. - Can attend 10 AM PDT call refer etherpad - https://etherpad.opendev.org/p/interop If you have any additional topics to discuss at PTG please add them to https://etherpad.opendev.org/p/interop_virtual_ptg_planning_june_2020 The conference Bridge will be assigned once we know the number of participants. Please add your names/topics to ether pad interop ptg planning. Optionally reply to this email with your inputs/patches. Please Register to attend the Interop WG slot scheduled for Virtual PTG on June 1 6-8 AM PDT / 9-11 AM EST / 14-16 UTC / 9-11 PM BJT https://www.openstack.org/ptg/ Thanksfor Interop WGChair - Prakash RamchandranVice Chair - Mark Voelker

4 4

Bug deputy report for week of 2020-05-18
by Maciej Jozefczyk 25 May '20

25 May '20

Hello, Hey, Here's the bug deputy report for last week: ===== High ===== * https://bugs.launchpad.net/neutron/+bug/1879301 [OVN] networking-ovn-tempest-dsvm-ovs-release-python2 job starts to fail on tempest py2 installation fix already merged: https://review.opendev.org/#/c/728817/ * https://bugs.launchpad.net/neutron/+bug/1879307 Recent stable/rocky change breaks networking-calico's interface driver fix proposed: https://review.opendev.org/#/c/729167/ * https://bugs.launchpad.net/neutron/+bug/1879950 [OVN] neutron-ovn-db-sync-util wipes out hash ring nodes fix proposed: https://review.opendev.org/#/c/729997/ ===== Medium ===== * https://bugs.launchpad.net/neutron/+bug/1879215 [L3] Unexcepted HA router scheduled instance shows up after manully scheduling Unassigned. * https://bugs.launchpad.net/neutron/+bug/1879502 [ovn] neutron-ovn-db-sync-util fails with PortContext has no session fix proposed: https://review.opendev.org/#/c/729264/ * https://bugs.launchpad.net/neutron/+bug/1879747 [DOC] Manual install & Configuration in neutron fix proposed: https://review.opendev.org/#/c/729894/ ===== Low ===== * https://bugs.launchpad.net/neutron/+bug/1879407 [OVN] Modifying FIP that is no associated causes ovn_revision_numbers to go stale unassigned, no big deal * https://bugs.launchpad.net/neutron/+bug/1879716 Policy is not enforced for network mtu config unassigned ==== Reopened ==== https://bugs.launchpad.net/neutron/+bug/1742401 Fullstack tests neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork fails often ===== Incomplete ===== * https://bugs.launchpad.net/neutron/+bug/1878719 DHCP Agent's iptables CHECKSUM rule causes skb_warn_bad_offload kernel * https://bugs.launchpad.net/neutron/+bug/1878905 when I use dvr mode for neutron,it's not functional * https://bugs.launchpad.net/neutron/+bug/1879009 attaching extra port to server raise duplicate dns-name error fix proposed: https://review.opendev.org/#/c/729440/ (in progress) See you, Maciej -- Best regards, Maciej Józefczyk

1 0

[neutron] Team meeting on Monday 25.05.2020 cancelled
by Slawek Kaplonski 25 May '20

25 May '20

Hi, As today is the Memorial Day in the USA, and many from team members are on holidays, lets cancel today's team meeting. Next week there is virtual PTG so we will meet there :) -- Slawek Kaplonski Senior software engineer Red Hat

1 0

[swift] Rolling upgrade, any version relationships?
by Mark Kirkwood 25 May '20

25 May '20

Hi, we are looking at upgrading our 2.7.0 Swift cluster. In the past I've modeled this on a dev system by upgrading storage nodes one by one (using 2.17 as the target version). This seemed to work well - I deliberately left the cluster half upgraded for an extended period to test for any cross version weirdness (didn't see any). However I'm wanting to check that I have not missed something important. So my questions are: - If upgrading from 2.7.0 is it safe to just grab the latest version (e.g 2.23)? - If not is there a preferred version to jump to first? - Is it ok for the upgrade to take extended time (e.g weeks) and therefore be running with some new and some old storage nodes for that time? regards Mark

3 5

[keystone][devstack][openstackclient] Problem with Keystone setup in stable/ussuri devstack install
by Neil Jerram 23 May '20

23 May '20

+ ./stack.sh:main:1091 : create_keystone_accounts + lib/keystone:create_keystone_accounts:314 : local admin_project ++ lib/keystone:create_keystone_accounts:315 : oscwrap project show admin -f value -c id WARNING: Failed to import plugin identity. ++ functions-common:oscwrap:2370 : return 2 + lib/keystone:create_keystone_accounts:315 : admin_project='openstack: '\''project show admin -f value -c id'\'' is not an openstack command. See '\''openstack --help'\''. I believe this is a completely mainline Keystone setup by devstack. The top-level stack.sh code is echo_summary "Starting Keystone" if [ "$KEYSTONE_AUTH_HOST" == "$SERVICE_HOST" ]; then init_keystone start_keystone bootstrap_keystone fi create_keystone_accounts ... bootstrap_keystone succeeded but create_keystone_accounts failed as shown above, trying to execute openstack project show "admin" -f value -c id IIUC, the rootmost problem here is "WARNING: Failed to import plugin identity.", indicating that python-openstackclient is failing to import its openstackclient.identity.client module. But I don't know any more about why that would be. Any ideas? Many thanks, Neil

2 3

[all][qa] Gate Status: week 20-21
by Ghanshyam Mann 23 May '20

23 May '20

Hello Everyone, I am highlighting the few of the gate issues we fixed or in-progress recently in case you are facing the same in your projects gate or 3rd party CI. 1. Bug# 1880219. A job using older Tempest (ubuntu xenial node jobs) on stable stein onwards gate or 3rd party CI fail. This is failing the same type of job I reported a few weeks back but with different issues and happening on 3rd party CI as I have not seen such a job or failure on upstream yet. I am reproducing the scenario to test the fixes[1]. Earlier issue[2] was fixed by using the older Tempest. Now it started failing as we use the master constraint on stein and train and master constraints are no more compatible with older Tempest[3]. The solution for this will be to use stable constraints if older Tempest is not used. This fix needs modification on devstack as well as the tempest side. Devstack changes[4] working fine but I need to fix it on the Tempest side also. 2. Bug#1880217. In-tree Tempest plugins python2 jobs on stable/stein or Train failing This might be a rare case but still possible as we do have in-tree plugins for older stable branches. For in-tree plugins (in case it's still present even we migrated most of the tempest plugins into separate repo ), 'all-plugins' tox env which is supposed to detect the system-wide installed plugins will not work now for python2.7 jobs. This is because Tempest is now py3-only and so create py3 venv even on py2 jobs. 'all-plugins' venv which is py3 no longer detect the tempest plugins installed as python2.7 site-packages. Failure example: https://zuul.opendev.org/t/openstack/build/7d3f36f464bb4642bb1647f4b0b8e91f… If you encounter this issue then a quick fix for that will be to install the in-tree plugins in Tempest venv via TEMPEST_PLUGINS and run tests with 'all' tox venv. Example: https://review.opendev.org/#/c/730378/2 3. hacking fix for new flake8 3.8.0 merged in master might need to be backported to stable branches also but not for all projects. But to have a safe guard against future flake8 3.9.0(which can also cause same issue), you can backport the hacking min version fix: - http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014828.html Have a happy weekend and feel free to append more gate bugs here you have encountered or fixed recently. [1] https://review.opendev.org/#/c/729611/3 [2] http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014601.html [3] https://review.opendev.org/#/c/720264/ [4] https://review.opendev.org/#/c/729722/6 -gmann

1 0

[swift] Erasure code hardware offloading
by Mark Kirkwood 22 May '20

22 May '20

Hi, Can Swift make use of an Erasure Code offload card (e.g https://community.mellanox.com/s/article/understanding-erasure-coding-offlo… regards Mark

2 1

[charms] Propose Aurelien Lourot for OpenStack Charms core
by Frode Nordahl 22 May '20

22 May '20

Hello all, I would like to propose adding Aurelien Lourot as core developer for the Charms project. -- Frode Nordahl

4 3

[release] Release countdown for week R-20, May 25 - 29
by Sean McGinnis 22 May '20

22 May '20

Welcome back to the release countdown emails! These will be sent at major points in the Victoria development cycle, which should conclude with a final release on October 16, 2020. Development Focus ----------------- At this stage in the release cycle, focus should be on planning the Victoria development cycle, assessing Victoria community goals and approving Victoria specs. General Information ------------------- Victoria is a 22 week development cycle, which is already underway. In case you haven't seen it yet, please take a look over the schedule for this release: https://releases.openstack.org/victoria/schedule.html By default, the team PTL is responsible for handling the release cycle and approving release requests. This task can (and probably should) be delegated to release liaisons. Now is a good time to review release liaison information for your team and make sure it is up to date: https://opendev.org/openstack/releases/src/branch/master/data/release_liais… By default, all your team deliverables from the Ussuri release are continued in the Victoria series with a similar release model. If you intend to drop a deliverable, or modify its release model, please do so before the victoria-1 milestone by proposing a change to the deliverable file at: https://opendev.org/openstack/releases/src/branch/master/deliverables/victo… Upcoming Deadlines & Dates -------------------------- Virtual Victoria PTG: June 1-5 Victoria-1 milestone: June 18

1 0

[neutron][operators]
by Dincer Celik 22 May '20

22 May '20

Hi, You know in public cloud world, most users prefer using VPCs (aka tenant networks for us), and when they need to connect two separate VPC, they have VPC peering functionality. For OpenStack, we have routers, RBACs, static routes and floating IPs, but there is nothing functioning like a real VPC peering. I see this was discussed at Stein PTG[1] under L3 topics but seems there is no progress about it. So I would like to ask what is the most common way to let two separate tenant networks under different accounts talk to each other? Thanks! Dincer [1] https://etherpad.opendev.org/p/neutron-stein-ptg

1 0

[charms] OpenStack Charms 20.05 release is now available
by Peter Matulis 22 May '20

22 May '20

The OpenStack Charms team is happy to announce the 20.05 charms release, introducing support for OpenStack Ussuri and Ceph Octopus on Ubuntu 18.04 LTS and Ubuntu 20.04 LTS. This release also brings several new and valuable features to the existing OpenStack Charms deployments for Queens, Rocky, Stein, Train, Ussuri and many other stable combinations of Ubuntu + OpenStack. Please see the Release Notes for full details: https://docs.openstack.org/charm-guide/latest/2005.html == Highlights == * OpenStack Ussuri OpenStack Ussuri is now supported on Ubuntu 18.04 LTS (via UCA) and Ubuntu 20.04 LTS natively. * Ceph Octopus The Octopus release of Ceph is now supported, starting with OpenStack Ussuri. * New charms: MySQL 8 The MySQL 8 charms (mysql-innodb-cluster and mysql-router) have been promoted to supported status. These charms allow for a completely native cloud database HA cluster (no need for the hacluster charm). The legacy solution (percona-cluster charm) is not supported on Ubuntu 20.04 LTS. Migration documentation is available. * New charms: Masakari The Masakari charms (masakari and masakari-monitors) and the requisite pacemaker-remote charm have been promoted to supported status, starting with OpenStack Stein. Masakari provides automated recovery of OpenStack instances (shared storage required). * New charms: OVN The OVN charms (neutron-api-plugin-ovn, ovn-central, ovn-chassis, and ovn-dedicated-chassis) have been promoted to supported status. OVN provides virtual networking for Open vSwitch and is the preferred default for new deployments of OpenStack Ussuri. * Ceph iSCSI Ceph iSCSI gateway functionality is available as a tech preview. These gateways provide iSCSI targets backed by a Ceph cluster. == OpenStack Charms team == The OpenStack Charms team can be contacted on the #openstack-charms IRC channel on Freenode. == Thank you == Lots of thanks to the below 67 charm contributors who squashed 89 bugs, enabled support for a new release of OpenStack, improved documentation, and added exciting new functionality! James Page Alex Kavanagh Frode Nordahl Liam Young Peter Matulis David Ames Corey Bryant Chris MacNaughton Ghanshyam Mann Aurelien Lourot Sahid Orentino Ferdjaoui Stamatis Katsaounis Ryan Beisner inspurericzhang Tiago Pasqualini Andrew McLeod ShangXiao Edward Hope-Morley Felipe Reyes kangyufei Adam Dyess Andreas Jaeger wangfaxin Xuan Yandong Arif Ali José Pekkarinen Chris Johnston Hemanth Nakkina Alexandros Soumplis Jose Guedez Qitao Tytus Kurek Seyeong Kim Dmitrii Shcherbakov Dongdong Tao Drew Freiberger Haw Loeung Jorge Niedbalski Andrea Ieri Xav Paice Qiu Fossen Yanos Angelopoulos Syed Mohammad Adnan Karim JiaSiRui Sérgio Filipe Marques Manso Vladimir Grevtsev George Kraft Marco Filipe Moutinho da Silva Xiyue Wang Adam Dyess Jose Delarosa Alexander Balderson Facundo Ciccioli Jacek Nykis Eduardo Sousa Thobias Trevisan Rodrigo Barbieri Alejandro Santoyo Gonzalez Sean McGinnis Claudio Pisa Shuo Liu Matus Kosut Stephen Muss Camille Rodriguez zhangboye Aggelos Kolaitis Nicolas Bock -- OpenStack Charms Team

2 1

[neutron] Drivers meeting 22.05.2020
by Slawek Kaplonski 22 May '20

22 May '20

Hi, Due to lack of agenda for the next drivers meeting, lets cancel it this week. See You all next week. BTW. Please maybe check Nate's email [1] - this isn't exactly RFE but something which IMO drivers team should check too :) [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014820.html -- Slawek Kaplonski Senior software engineer Red Hat

1 0

[kolla] Kolla klub meeting
by Mark Goddard 21 May '20

21 May '20

Hi, Just a reminder that we will host a project onboarding session for the next Kolla Klub meeting on Thursday 21st May. We'll cover a variety of topics on the many different ways you can contribute to the project. Gaël Therond will also fill us in on some of the questions put to him since the last meeting about his case studies. https://docs.google.com/document/d/1EwQs2GXF-EvJZamEx9vQAOSDB5tCjsDCJyHQN5_… Thanks, Mark

1 1

review of tripleo-ci-centos-8-standalone-on-multinode-ipa
by Wesley Hayutin 21 May '20

21 May '20

Greetings, Please review the following new standalone + ipa deployment [1]. The keystone folks would like this deployment to run by default on all THT changes vs. the current set of files [2]. Looking for any strong objects to running the job on any THT change and or more files to target the job against in [2] If you need specific information about the job please contact rlandy @ redhat pojadhav @ redhat Thanks [1] https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-o… [2] https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/zuul…

1 0

[Heat] Heat Stack Create Authorization Failed
by İzzettin Erdem 21 May '20

21 May '20

Hello everyone, When I launch a cluster on Sahara, it gives a heat stack authorization error. I also use Murano service and it is working with heat. I could not find the solution and I discuss this error with the Sahara-devel team. They are also searching for this. Could you help me, please? Both the error log of Sahara and Heat are below. Sahara-engine: http://paste.openstack.org/show/793671/ Heat-engine: http://paste.openstack.org/show/793673/ Thanks. Regards.

4 4

[ptg][cyborg][neutron][nova]Smartnic support integration via cyborg neutron and nova? - Cyborg Pre-PTG Meeting
by yumeng bao 21 May '20

21 May '20

Hi, As agreed on the IRC, tomorrow, we will have a cyborg Pre-PTG Zoom Meeting. Shaohe Feng from Intel will introduce SmartNIC support from the following aspects: * the cyborg-neutron-nova integration workflow when creating a server with a port of SmartNIC which is managed by cyborg *what changes needs to be done in nova, neutron to support SmartNIC feature? Please find the Pre-PTG meeting Link below: Zoom URL: https://us04web.zoom.us/j/9575741145?pwd=cVZydHl6blYzTGtNVlU4QU96ZWxiZz09 Meeting ID：957 574 1145 Password：9XhFhK The reserved meeting time is in total one and a half hours, but divided into three time slots(if time is not enough, we can extend another half hour time slot). *May 22 Friday 2:30 UTC -3:00 UTC *May 22 Friday 3:00 UTC -3:30UTC *May 22 Friday 3:30 UTC -4:00UTC * *And Please be aware: The three slots use the same URL, same Meeting ID and same Password. This means, when time of the slot is running out,and if you're kicked out of the Zoom, then you will need to login to the same Zoom URL again to join the next time slot. (And there’s also possibility that we don’t need to do the re-login, this depends on the number of people.） For others interested but not quite familiar with this topic, here’s some background information: Cyborg, served as a general purpose management framework for acceleration resources, is trying to support various types of accelerators.(FPGA, GPU, SPDK and so on) We continuously got questions from operators if cyborg supports SmartNIC, and this motivated us to support SmartNIC. Actually, we’ve made a discussion on SmartNIC with neutron team at the Shanghai Summit[1] 2019. We are now planning to continue discussing SmartNICs support integration in the coming Virtual PTG and try to move forward in Victoria Release. Cyborg will discuss about this topic internally first, and then later discuss with nova and neutron at PTG cross-project session[2][3]. [1] https://etherpad.opendev.org/p/Shanghai-Neutron-Cyborg-xproj [2] https://etherpad.opendev.org/p/nova-victoria-ptg [3] https://etherpad.opendev.org/p/cyborg-victoria-goals Regards， Yumeng

1 0

[nova][cinder][linstor] VM creation failed with invalid libvirt xml <source dev='None'/>
by Volodymyr Litovka 21 May '20

21 May '20

Hi colleagues, does anybody here use Linstor as backend for Openstack? I can not launch VM (synopsis below) while everything looks fine until the last step :-) Few words about Openstack's storage configuration: - controllers: cinder-api, cinder-scheduler, cinder-volume (modified with latest driver from Linstor's git), linstor-client, python-linstor - compute nodes: linstor-controller, linstor-satellite, linstor-client, python-linstor; node types - Combined - cinder.conf[1] - volume-type registered with corresponding properties[2] I'm able to manipulate volumes - create/delete[3] and every existing volume has the corresponding drbd/linstor repr[4] and LVM LV[5] on the compute nodes, so, from storage point of view, it looks as correct and ready to use. The problem is when I try to create VM using $ openstack server create --flavor [...] --network [...] --volume 15c4fa98-76a5-44d6-9701-c89a8a848b82 fearst Nova fails[6] because of invalid VM's libvirt XML definition (note "source dev='None'" below): <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='None'/> <target dev='vda' bus='virtio'/> <serial>15c4fa98-76a5-44d6-9701-c89a8a848b82</serial> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> I do not have another working DRBD/Linstor/Openstack installation and can't even imagine, where to look into the problem - whether it is in Nova, Cinder or in Linstor configuration itself. I will appreciate if anybody can give me some guidance on where to seek for the solution to the problem. In my lab I'm using Openstack Train on Ubuntu 18.04.4: nova-compute 2:20.1.1-0ubuntu1~cloud0 nova-compute-kvm 2:20.1.1-0ubuntu1~cloud0 nova-compute-libvirt 2:20.1.1-0ubuntu1~cloud0 nova-api 2:20.1.1-0ubuntu1~cloud0 nova-common 2:20.1.1-0ubuntu1~cloud0 nova-conductor 2:20.1.1-0ubuntu1~cloud0 nova-novncproxy 2:20.1.1-0ubuntu1~cloud0 nova-scheduler 2:20.1.1-0ubuntu1~cloud0 cinder-api 2:15.1.0-0ubuntu1~cloud0 cinder-common 2:15.1.0-0ubuntu1~cloud0 cinder-scheduler 2:15.1.0-0ubuntu1~cloud0 cinder-volume 2:15.1.0-0ubuntu1~cloud0 python3-cinder 2:15.1.0-0ubuntu1~cloud0 python3-cinderclient 1:5.0.0-0ubuntu2~cloud0 python3-nova 2:20.1.1-0ubuntu1~cloud0 python3-novaclient 2:15.1.0-0ubuntu2~cloud0 Thank you! ==================================== [1] cinder.conf [DEFAULT] enabled_backends = linstor default_volume_type = linstor [linstor] storage_availability_zone = nova volume_backend_name = linstor volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver linstor_autoplace_count = 2 linstor_default_volume_group_name=DfltRscGrp linstor_default_uri=linstor://runner linstor_default_storage_pool_name=drbdpool linstor_default_resource_size=1 linstor_volume_downsize_factor=4096 where 'runner' is virtual ip of linstor-controller, protected by pacemaker. ==================================== [2] openstack volume type show linstor +--------------------+--------------------------------------+ | Field | Value | +--------------------+--------------------------------------+ | access_project_ids | None | | description | None | | id | d2025962-503a-4f37-93bd-b766bb346a42 | | is_public | True | | name | linstor | | properties | volume_backend_name='linstor' | | qos_specs_id | None | +--------------------+--------------------------------------+ ==================================== [3] openstack volume show fearst +--------------------------------+-----------------------------------------------------+ | Field | Value | +--------------------------------+-----------------------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | true | | consistencygroup_id | None | | created_at | 2020-05-21T08:42:11.000000 | | description | None | | encrypted | False | | id | 15c4fa98-76a5-44d6-9701-c89a8a848b82 | | migration_status | None | | multiattach | False | | name | fearst | | os-vol-host-attr:host | ctrl1@linstor#linstor | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 7acec404393344cbabde07a22bbe6b3f | | properties | | | replication_status | None | | size | 5 | | snapshot_id | None | | source_volid | None | | status | available | | type | linstor | | updated_at | 2020-05-21T08:42:18.000000 | | user_id | 1048a19e94234451a61c3d9d46907ceb | | volume_image_metadata | {'signature_verified': 'False', | | | 'image_id': '15e992e8-a601-44cd-a2f0-8f34588a4d18',| | | 'image_name': 'cirros', | | | 'checksum': '1d3062cd89af34e419f7100277f38b2b', | | | 'container_format': 'bare', | | | 'disk_format': 'qcow2', | | | 'min_disk': '0', | | | 'min_ram': '0', | | | 'size': '16338944'} | +--------------------------------+-----------------------------------------------------+ ==================================== [4] linstor r l ╭───────────────────────────────────────────────────────────────────────────────────╮ ┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ ╞═══════════════════════════════════════════════════════════════════════════════════╡ ┊ CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 ┊ cmp1 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ ┊ CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 ┊ cmp2 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ ╰───────────────────────────────────────────────────────────────────────────────────╯ Note, that cmp3 is arbiter and has no 'by-disk' representation: root@cmp1:~# ls -lR /dev/drbd* brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000 /dev/drbd/by-disk/sds: lrwxrwxrwx 1 root root 17 May 21 11:42 CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 -> ../../../drbd1000 /dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82: lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000 root@cmp2:~# ls -lR /dev/drbd* brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000 /dev/drbd/by-disk/sds: lrwxrwxrwx 1 root root 17 May 21 11:42 CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 -> ../../../drbd1000 /dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82: lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000 root@cmp3:~# ls -lR /dev/drbd* brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000 /dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82: lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000 ==================================== [5] root@cmp1:~# lvdisplay [ ... ] --- Logical volume --- LV Path /dev/sds/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 LV Name CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 VG Name sds LV UUID jGzcIQ-yRpH-RKxP-ExsH-JSn6-kTFn-8kpKYO LV Write Access read/write LV Creation host, time cmp1, 2020-05-21 11:42:14 +0300 LV Pool name thin LV Status available # open 2 LV Size 5.00 GiB Mapped size 0.02% Current LE 1280 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:4 root@cmp2:~# lvdisplay [ ... ] --- Logical volume --- LV Path /dev/sds/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 LV Name CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 VG Name sds LV UUID 8pVwMo-e3ed-Sy0w-06IZ-M7Mh-YVIL-BtKZEG LV Write Access read/write LV Creation host, time cmp2, 2020-05-21 11:42:14 +0300 LV Pool name thin LV Status available # open 2 LV Size 5.00 GiB Mapped size 0.02% Current LE 1280 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:4 ==================================== [6] Traceback of nova-compute: 3304 ERROR nova.compute.manager [] Traceback (most recent call last): 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2419, in _build_and_run_instance 3304 ERROR nova.compute.manager [] block_device_info=block_device_info) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3474, in spawn 3304 ERROR nova.compute.manager [] power_on=power_on) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6261, in _create_domain_and_network 3304 ERROR nova.compute.manager [] destroy_disks_on_failure) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 3304 ERROR nova.compute.manager [] self.force_reraise() 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 3304 ERROR nova.compute.manager [] six.reraise(self.type_, self.value, self.tb) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise 3304 ERROR nova.compute.manager [] raise value 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6230, in _create_domain_and_network 3304 ERROR nova.compute.manager [] post_xml_callback=post_xml_callback) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6164, in _create_domain 3304 ERROR nova.compute.manager [] guest.launch(pause=pause) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 143, in launch 3304 ERROR nova.compute.manager [] self._encoded_xml, errors='ignore') 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 3304 ERROR nova.compute.manager [] self.force_reraise() 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 3304 ERROR nova.compute.manager [] six.reraise(self.type_, self.value, self.tb) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise 3304 ERROR nova.compute.manager [] raise value 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 138, in launch 3304 ERROR nova.compute.manager [] return self._domain.createWithFlags(flags) 3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 190, in doit 3304 ERROR nova.compute.manager [] result = proxy_call(self._autowrap, f, *args, **kwargs) 3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 148, in proxy_call 3304 ERROR nova.compute.manager [] rv = execute(f, *args, **kwargs) 3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 129, in execute 3304 ERROR nova.compute.manager [] six.reraise(c, e, tb) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise 3304 ERROR nova.compute.manager [] raise value 3304 ERROR nova.compute.manager [] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 83, in tworker 3304 ERROR nova.compute.manager [] rv = meth(*args, **kwargs) 3304 ERROR nova.compute.manager [] File "/usr/lib/python3/dist-packages/libvirt.py", line 1110, in createWithFlags 3304 ERROR nova.compute.manager [] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 3304 ERROR nova.compute.manager [] libvirt.libvirtError: Cannot access storage file 'None': No such file or directory -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison

1 0

[neutron][release] Missing tarballs for networking-mlnx, networking-l2gw and vmware-nsx
by Mark Goddard 21 May '20

21 May '20

Hi, There are no tarballs published for ussuri networking-mlnx or vmware-nsx. The most recently published tarballs were in February. Is this expected? networking-l2gw also has no ussuri tarballs, although master was updated recently. Thanks, Mark

7 8

[kolla] deprecating kolla-cli
by Mark Goddard 21 May '20

21 May '20

Hi, We previously planned to retire kolla-cli due to lack of interest, but at the last minute there was an offer to help support it. That support is no longer available, and we are not aware of the project having gained traction since our original decision. We are therefore adding a deprecation notice [1] to the Ussuri release notes, with a plan to remove it from governance in Victoria if there is no response to the call for help. [1] https://review.opendev.org/729855 Thanks, Mark

1 0

[keystone][ldap]
by Amjad Kotobi 20 May '20

20 May '20

Hi all, I’m integrating keystone with LDAP, and having “service account” e.g. Nova, keystone etc.. which are in database. As soon as connecting it to ldap all authentication getting failed, how can I have both “service account” and “LDAP users” connected to Keystone? Here is my keystone.conf ################### [ldap] url = ldap://XXXXX user = uid=XXX,cn=sysaccounts,cn=etc,dc=XXX,dc=de password = dkrzprox user_tree_dn = cn=users,cn=accounts,dc=XXX,dc=de user_objectclass = posixAccount user_id_attribute = uid user_name_attribute = uid user_allow_create = false user_allow_update = false user_allow_delete = false group_tree_dn = cn=groups,cn=accounts,dc=XXX,dc=de group_objectclass = groupOfNames group_id_attribute = cn group_name_attribute = cn group_member_attribute = member group_desc_attribute = description group_allow_create = false group_allow_update = false group_allow_delete = false use_pool = true use_auth_pool = true debug_level = 4095 query_scope = sub [identity] driver = ldap ##################### OS: Centos7 OpenStack-Release: Train Any idea or example of options gonna be great! Thank you

3 2

[keystone]
by Amjad Kotobi 20 May '20

20 May '20

Hi, I have keystone running in HA multiple nodes, and after reboot one of the node it started in logs to pop up below messages “""WARNING keystone.server.flask.application [req-fe993224-cc7e-4e5f-83a8-1e925e60b995 24dfa0beaaaa48848b4eee68e449f2de d1fecbdb49ef4871b94b201b7856eed3 - default default] Could not recognize Fernet token: TokenNotFound: Could not recognize Fernet token””” And in command line “”"Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-1145c696-6275-4c78-9486-a690b98548ed)””" During command lines when request lands on machine which hasn’t rebooted or so, the result comes but on the rest fore-mentioned message always in place. Openstack release Train, and deployed manually. How can I tackle it down? Any ideas? Thanks Amjad

2 1

[docs][all] Important changes in recent openstackdocstheme updates
by Andreas Jaeger 20 May '20

20 May '20

A couple of changes recently merged into openstackdocstheme to fix problems reported. These had some surprises in it and we'd like to inform you about the changes: * Config options are now prefixed with openstackdocs_, the old names will be removed in a future release * The 'project' config option is now only respected (and displayed in the left menu) if 'openstackdocs_auto_name = False' is set. By default, the theme uses the package name (from setup.cfg) * The HTML files show the version number by default (with exception of releasenotes and api docs) calculated from git. If you want to use your own version number or disable it, set 'openstackdocs_auto_version = False' and manually configure the 'version' and 'release' options. * Previously, the theme always used 'pygments_style = "native"' and overrode the setting of 'sphinx' that many repos have. Now the setting is respected. For a few repos this lead to unreadable code snippets. If you see this or want to go back to the previous theme, configure 'pygments_style = "native"'. * Many projects have written PDF documents. openstackdocstheme can now optionally link to them. Set 'openstackdocs_pdf_link' to True to show the icon with path. Note that the PDF file is placed on docs.openstack.org in the top of the html files while in check/gate it's in a separate PDF folder. Thus, the site preview will show in check/gate a broken link - but it works fine, check [2]. * Both reno (since version 3.1.0) and openstackdocsstheme are now declared parallel safe, the CI jobs automatically build releasenotes in parallel [1]. You can modify your local tox job to do this by adding the '-j auto' parameter to your 'sphinx-build' invocation. We're releasing openstackdocstheme version 2.2.1 soon with two further fixes: * PDF documents will now show the version number like html document, no need to configure versions in conf.py for this anymore [3]. * small bug fix (if you set auto_name = False in doc/source/conf.py, this hit so far 5 repos)[4]. Everything is documented in the documentation of openstackdocstheme [2]. If there are any questions, best ask in #openstack-oslo. Andreas has started pushing changes to update projects with topic:reno-openstackdocstheme. Hope that's all for Victoria on the openstackdocstheme, Stephen and Andreas [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014902.html [2] https://docs.openstack.org/openstackdocstheme/ [3] https://review.opendev.org/729554 [4] https://review.opendev.org/729031 -- Andreas Jaeger aj(a)suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB

1 0