May 2020 - openstack-discuss - lists.openstack.org

[tripleo] TripleO Ussuri has been released
by Wesley Hayutin 27 May '20

27 May '20

Greetings, I'm pleased to announce that TripleO Ussuri has been released today. Special thanks to Chandan Kumar and Emilien Macchi for the extra time and effort in getting everything in place for the release. Thanks to everyone else as well for your contributions, bug reports, patches and participation! Some highlights from the TripleO release: * The mistral to ansible conversion blueprint is complete * The nova-less-deploy feature is complete * The TripleO-Operator-Ansible feature is complete * The RDO and CI teams completed work on component based DLRN rpm repos and CI. * TLS and TripleO deployments tested together in upstream CI. * Support for CentOS-8 * Many other improvements and updates Congratulations to the following individual contributors for their reviews across ALL openstack projects!! Alex Schultz, the #1 reviewer for all projects in Ussuri Herve Beraud, the #3 reviewer for all projects in Ussuri Emilien Macchi, the #4 reviewer for all projects in Ussuri Followed closely behind in reviews by Stephen Finucane Dmitry Tantsur Slawek Kaplonski Thanks to all our TripleO contributors!! https://www.stackalytics.com/?release=ussuri&metric=commits&project_type=op… Oh by the way TripleO fixed 558 bugs this release as well :) Additional statistics can be found here: https://docs.google.com/presentation/d/1RV30OVxmXv1y_z33LuXMVB56TA54Urp7oHI… <https://docs.google.com/presentation/d/1RV30OVxmXv1y_z33LuXMVB56TA54Urp7oHI…> Release notes can be found per TripleO project: https://docs.openstack.org/releasenotes/ At the time of writing this email not all the release notes have been published. Thanks all and I'm looking forward to seeing you at the PTG!

2 1

[PTG][neutron] Draft of the agenda
by Slawek Kaplonski 26 May '20

26 May '20

Hi neutrinos, I just prepared draft of the agenda for the Neutron team for the upcoming virtual PTG. It's available in the etherpad [1]. Please take a look into it and reach out to me if You: * want to change day/time of topic(s) which You are interested in mostly due to e.g. some conflicts which You may have with some other meetings, * if You want to add some new topics which, to make sure that agenda is modified to include that, * if You see anything else which You thinks that should be changed there :) See You all on the virtual PTG next week :) [1] https://etherpad.opendev.org/p/neutron-victoria-ptg -- Slawek Kaplonski Senior software engineer Red Hat

1 0

[all][policy] Secure Default Policies Popup team meeting
by Ghanshyam Mann 26 May '20

26 May '20

Hello Everyone, To speed up the work on Secure Default Policies Popup team[1], we are going to start the biweekly meeting to discuss the pending things to finish if any or project need help on implementing the new default policy. We encourage projects interested to implement the new defaults or at least those listed on the wiki page [2] to join that meeting. Meeting details are on the wiki[3] also I have proposed the patch to book the slot: https://review.opendev.org/#/c/730935/ Agenda is slightly open for now which we will be moving to concrete as we progress and timing is biweekly-even on Thursday starting from this week on 28th 1800 UTC. We can adjust the timing also if needed. NOTE: First meeting on this week Thursday 28th 1800UTC on #openstack-meeting [1] https://governance.openstack.org/tc/reference/popup-teams.html#secure-defau… [2] https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… [3] https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… -gmann

1 0

[openstack-helm] Stepping down as core reviewer
by Steve Wilkerson 26 May '20

26 May '20

Hey everyone, I hope you're all staying safe and well. I switched jobs a few months back that took me in a different direction from my previous employer, and as a result I've not had the time to remain involved with the OpenStack community (especially as my day job no longer involves working with OpenStack). I think it's best that I step down from the openstack-helm core reviewer team, as I can't realistically make the time to do my due diligence in providing thorough, useful code reviews. I greatly appreciate the opportunities that working with the OpenStack community at large has provided me over the past 5 years. Cheers, srwilkers

2 1

[kolla-ansible][kolla][cinder]Cinder volume is not attached to instance
by Sanjay K 26 May '20

26 May '20

Hello Openstackers, I have used Cinder stein release (v14.0.2) source base and deployed in All-In-One setup on ubuntu16.04 using Kolla-Ansible. The cinder image was built using python 2.7 somewhere which i am trying to deploy in my new setup having python3.5. All services are deployed and running fine. However when I try to attach one volume to an instance, it shows below error in /var/log/kolla/cinder/cinder-volume.log. 2020-05-26 08:10:07.026 35 INFO oslo.privsep.daemon [req-db287360-876c-425e-b8df-a4f3b0db3163 bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'cinder.privsep.sys_admin_pctxt', '--privsep_sock_path', '/tmp/tmpan8c8r/privsep.sock'] 2020-05-26 08:10:10.952 35 INFO oslo.privsep.daemon [req-db287360-876c-425e-b8df-a4f3b0db3163 bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Spawned new privsep daemon via rootwrap 2020-05-26 08:10:10.647 1071 INFO oslo.privsep.daemon [-] privsep daemon starting 2020-05-26 08:10:10.721 1071 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2020-05-26 08:10:10.725 1071 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_NET_ADMIN|CAP_SYS_ADMIN/none 2020-05-26 08:10:10.726 1071 INFO oslo.privsep.daemon [-] privsep daemon running as pid 1071 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server [req-db287360-876c-425e-b8df-a4f3b0db3163 bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Exception during message handling: ProcessExecutionError: Unexpected error while running command. Command: tgtadm --lld iscsi --op show --mode target Exit code: 107 Stdout: u'' Stderr: u'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 4517, in attachment_update 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server connector) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 4445, in _connection_create 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server volume, connector) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 830, in create_export 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server volume_path) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 210, in create_export 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server **portals_config) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/utils.py", line 818, in _wrapper 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 206, in call 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return attempt.get(self._wrap_exception) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 247, in get 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2]) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 200, in call 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/targets/tgt.py", line 126, in create_iscsi_target 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server (out, err) = cinder.privsep.targets.tgt.tgtadm_show() 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 244, in _wrap 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 203, in remote_call 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server raise exc_type(*result[2]) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command. 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Command: tgtadm --lld iscsi --op show --mode target 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Exit code: 107 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Stdout: u'' 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Stderr: u'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server When I check inside cinder-volume container, the tgtd is not running. SO I started tgtd from root user. Now the volume attachment happens for few seconds and then it is detached. 2020-05-26 08:14:39.221 35 INFO cinder.volume.manager [req-5da81388-33ce-4994-9114-52442a08ac2a bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] attachment_update completed successfully. 2020-05-26 08:14:52.622 35 INFO cinder.volume.manager [req-cbaa1b6c-4fa8-4ece-b08b-8ca09f78ecaf bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Terminate volume connection completed successfully. 2020-05-26 08:14:52.840 35 INFO cinder.volume.targets.tgt [req-cbaa1b6c-4fa8-4ece-b08b-8ca09f78ecaf bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Removing iscsi_target for Volume ID: d713ce55-1fa6-4d96-a69e-69368494f8f9 2020-05-26 08:14:53.739 35 WARNING py.warnings [req-cbaa1b6c-4fa8-4ece-b08b-8ca09f78ecaf bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] /var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/evaluator.py:99: SAWarning: Evaluating non-mapped column expression 'updated_at' onto ORM instances; this is a deprecated use case. Please make use of the actual mapped columns in ORM-evaluated UPDATE / DELETE expressions. "UPDATE / DELETE expressions." % clause I also used the cinder 14.0.4 source tag to build the new image, but the same error persists. Can you please point me what is the issue in my test case or setup. Appreciate your help. best regards, Sanjay

2 1

[Horizon] Failing to configure the Instance Launch dialog
by Bernd Bausch 26 May '20

26 May '20

On an Ussuri Devstack, I would like to launch instances with ephemeral storage from Horizon and remove the volume storage switch from the instance launch dialog. Prior to Train, the dialog was configured with the LAUNCH_INSTANCE_DEFAULTS dictionary in /local_settings.py/. This is my dictionary; I set /create_volume /to /False /and /hide_create_volume /to /True/: LAUNCH_INSTANCE_DEFAULTS = { 'config_drive': False, 'create_volume': False, 'hide_create_volume': True, 'disable_image': False, 'disable_instance_snapshot': False, 'disable_volume': False, 'disable_volume_snapshot': False, 'enable_scheduler_hints': True, } Starting with Train, the dictionary is in /.../openstack_dashboard/defaults.py/, but the changes I made there have no effect. Adding the dictionary to /local_settings.py/ has no effect either. To test whether /defaults.py/ is used at all, I added a syntactically incorrect line, and indeed, the syntax error is logged in /horizon_error.log/. I wonder what I am doing wrong. I am convinced it worked like this on Stein. Any ideas? Bernd Bausch

2 3

[TripleO] External network on compute node
by Reza Bakhshayeshi 26 May '20

26 May '20

Hi all. The default way of compute node for accessing Internet if through undercloud. I'm going to assign an IP from External network to each compute node with default route. But the deployment can't assign an IP to br-ex and fails with: " raise AddrFormatError('invalid IPNetwork %s' % addr)", "netaddr.core.AddrFormatError: invalid IPNetwork ", Actually 'ip_netmask': '' is empty during deployment for compute nodes. I've added external network to compute node role: External: subnet: external_subnet and for network interface: - type: ovs_bridge name: bridge_name mtu: get_param: ExternalMtu dns_servers: get_param: DnsServers use_dhcp: false addresses: - ip_netmask: get_param: ExternalIpSubnet routes: list_concat_unique: - get_param: ExternalInterfaceRoutes - - default: true next_hop: get_param: ExternalInterfaceDefaultRoute members: - type: interface name: nic3 mtu: get_param: ExternalMtu use_dhcp: false primary: true Any suggestion would be grateful. Regards, Reza

2 6

[Nova][Scheduler] Reducing race-conditions and re-scheduling during creation of multiple high-ressources instances or instances with anti-affinity.
by Laurent Dumont 25 May '20

25 May '20

Hey everyone, We are seeing a pretty consistent issue with Nova/Scheduler where some instances creation are hitting the "max_attempts" limits of the scheduler. Env : Red Hat Queens Computes : All the same hardware and specs (even weight throughout) Nova : Three nova-schedulers This can be due to two different factors (from what we've seen) : - Anti-affinity rules are getting triggered during the creation (two claims are done within a few milliseconds on the same compute) which counts as a retry (we've seen this when spawning 40+ VMs in a single server group with maybe 50-55 computes - or even less 14 instances on 20ish computes). - We've seen another case where MEMORY_MB becomes an issue (we are spinning new instances in the same host-aggregate where VMs are already running. Only one VM can run per compute but there are no anti-affinity groups to force that between the two deployments. The ressource requirements prevent anything else from getting spun on those). - The logs look like the following : - Unable to submit allocation for instance 659ef90e-33b8-42a9-9c8e-fac87278240d (409 {"errors": [{"status": 409, "request_id": "req-429c2734-2f2d-4d2d-82d1-fa4ebe12c991", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on resource provider '35b78f3b-8e59-4f2f-8cad-eaf116b7c1c7'. The requested amount would exceed the capacity. ", "title": "Conflict"}]}) / Setting instance to ERROR state.: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f6d06cca-e9b5-4199-8220-e3ff2e5c2a41. - I do believe we are hitting this issue as well : https://bugs.launchpad.net/nova/+bug/1837955 - In all the cases where the Stacks creation failed, one instance was left in the Build state for 120 minutes and then finally failed. >From what we can gather, there are a couple of parameters that be be tweaked. 1. host_subset_size (Return X number of host instead of 1?) 2. randomize_allocation_candidates (Not 100% on this one) 3. shuffle_best_same_weighed_hosts (Return a random of X number of computes if they are all equal (instance of the same list for all scheduling requests)) 4. max_attempts (how many times the Scheduler will try to fit the instance somewhere) We've already raised "max_attempts" to 5 from the default of 3 and will raise it further. That said, what are the recommendations for the rest of the settings? We are not exactly concerned with stacking vs spreading (but that's always nice) of the instances but rather making sure deployments fail because of real reasons and not just because Nova/Scheduler keeps stepping on it's own toes. Thanks!

3 6

[all][interop] Please suggest capabilities to deprecate-remove and add to the OpenStack Powered Programs
by prakash RAMCHANDRAN 25 May '20

25 May '20

Hi all, Please help the Interop WG to review and finalize the draft version for the next interop guidelines. https://review.opendev.org/gitweb?p=openstack/interop.git;a=blob;f=2020.06.… Link for the DNS add-on: https://review.opendev.org/gitweb?p=openstack/interop.git;a=blob;f=add-ons/… Link for the Heat add-on: https://review.opendev.org/gitweb?p=openstack/interop.git;a=blob;f=add-ons/… This guidelines is intended to cover "Stein" , "Train", "Ussuri" and "Victoria" releases of OpenStack. We request that PTLs or Core team members please confirm any updates you may want to include or exclude in link above. The Projects currently covered under OpenStack Powered Programs include "Keystone", "Glance", "Nova", "Neutron", "Cinder", "Swift" and add-on programs "Designate", "Heat". A more human-readable view of the capabilities can be found in RefStack: https://refstack.openstack.org/#/guidelines We would like have feedback by next Interop call on Friday. - Can attend 10 AM PDT call refer etherpad - https://etherpad.opendev.org/p/interop If you have any additional topics to discuss at PTG please add them to https://etherpad.opendev.org/p/interop_virtual_ptg_planning_june_2020 The conference Bridge will be assigned once we know the number of participants. Please add your names/topics to ether pad interop ptg planning. Optionally reply to this email with your inputs/patches. Please Register to attend the Interop WG slot scheduled for Virtual PTG on June 1 6-8 AM PDT / 9-11 AM EST / 14-16 UTC / 9-11 PM BJT https://www.openstack.org/ptg/ Thanksfor Interop WGChair - Prakash RamchandranVice Chair - Mark Voelker

4 4

Bug deputy report for week of 2020-05-18
by Maciej Jozefczyk 25 May '20

25 May '20

Hello, Hey, Here's the bug deputy report for last week: ===== High ===== * https://bugs.launchpad.net/neutron/+bug/1879301 [OVN] networking-ovn-tempest-dsvm-ovs-release-python2 job starts to fail on tempest py2 installation fix already merged: https://review.opendev.org/#/c/728817/ * https://bugs.launchpad.net/neutron/+bug/1879307 Recent stable/rocky change breaks networking-calico's interface driver fix proposed: https://review.opendev.org/#/c/729167/ * https://bugs.launchpad.net/neutron/+bug/1879950 [OVN] neutron-ovn-db-sync-util wipes out hash ring nodes fix proposed: https://review.opendev.org/#/c/729997/ ===== Medium ===== * https://bugs.launchpad.net/neutron/+bug/1879215 [L3] Unexcepted HA router scheduled instance shows up after manully scheduling Unassigned. * https://bugs.launchpad.net/neutron/+bug/1879502 [ovn] neutron-ovn-db-sync-util fails with PortContext has no session fix proposed: https://review.opendev.org/#/c/729264/ * https://bugs.launchpad.net/neutron/+bug/1879747 [DOC] Manual install & Configuration in neutron fix proposed: https://review.opendev.org/#/c/729894/ ===== Low ===== * https://bugs.launchpad.net/neutron/+bug/1879407 [OVN] Modifying FIP that is no associated causes ovn_revision_numbers to go stale unassigned, no big deal * https://bugs.launchpad.net/neutron/+bug/1879716 Policy is not enforced for network mtu config unassigned ==== Reopened ==== https://bugs.launchpad.net/neutron/+bug/1742401 Fullstack tests neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork fails often ===== Incomplete ===== * https://bugs.launchpad.net/neutron/+bug/1878719 DHCP Agent's iptables CHECKSUM rule causes skb_warn_bad_offload kernel * https://bugs.launchpad.net/neutron/+bug/1878905 when I use dvr mode for neutron,it's not functional * https://bugs.launchpad.net/neutron/+bug/1879009 attaching extra port to server raise duplicate dns-name error fix proposed: https://review.opendev.org/#/c/729440/ (in progress) See you, Maciej -- Best regards, Maciej Józefczyk

1 0