April 2021 - openstack-discuss - lists.openstack.org

Re: [Release-job-failures] Pre-release of openstack/neutron for ref refs/tags/18.0.0.0rc2 failed
by Thierry Carrez 07 Apr '21

07 Apr '21

zuul(a)openstack.org wrote: > Build failed. > > - openstack-upload-github-mirror https://zuul.opendev.org/t/openstack/build/4b31dfd25e244e38a346e097c993c519 : SUCCESS in 54s > - release-openstack-python https://zuul.opendev.org/t/openstack/build/309d80630c8e4086b52a0d83062cd431 : SUCCESS in 4m 43s > - announce-release https://zuul.opendev.org/t/openstack/build/4bde576261db4943a1dd6610aa52f6be : POST_FAILURE in 8m 33s > - propose-update-constraints https://zuul.opendev.org/t/openstack/build/18de7b1f60044445a06db64da42bbcc3 : SUCCESS in 5m 20s We are missing logs, but it looks like the job actually succeeded at announcing the release: http://lists.openstack.org/pipermail/release-announce/2021-April/011022.html So this can be safely ignored. -- Thierry Carrez (ttx)

2 1

[neutron][nova] Port binding fails when creating an instance
by Maxime d'Estienne 07 Apr '21

07 Apr '21

Hello, I spent a lot of time troubleshooting my issue, which I described here : https://serverfault.com/questions/1058969/cannot-create-an-instance-due-to-… To summarize, when I want to create an instance, binding fails on compute node, the dhcp agent seems to give an ip to the VM but I have an error. I don't know where to dig, besides what I have done. Thanks a lot for your help !

2 5

[Neutron] How to provide internet access to tier 2 instance
by Bernd Bausch 07 Apr '21

07 Apr '21

I have a pretty standard single-server Victoria Devstack, where I created this network topology: public private backend | | | | /-------\ |-- I1 |- I2 |--|Router1|--| | | \-------/ | | | | /-------\ | | |--|Router2|--| | | \-------/ | | | | I1 and I2 are instances. My question: Is it possible to give I2 access to the external world to install software and download files? I don't need access **to** I2 **from** the external world. My unsuccessful attempt: After adding a static default route via Router1 to Router2, I can ping the internet from Router2's namespace, but not from I2. My guess is that Router1 ignores traffic from networks that are not attached to it. I don't have enough experience to understand the netfilter rules in Router1's namespace, and in any case, rather than tweaking them I need a supported method to give I2 internet access, or the confirmation that it is not possible. Thanks much for any insights and suggestions. Bernd

2 1

[cinder] Bug deputy report for week of 2021-04-07
by Sofia Enriquez 07 Apr '21

07 Apr '21

Hello, This is a bug report from 2021-03-31 to 2021-04-07. You're welcome to join the next Cinder Bug Meeting later today. Weekly on Wednesday at 1500 UTC in #openstack-cinder Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Critical:- High: - Medium: - https://bugs.launchpad.net/cinder/+bug/1922408 `` Create volume from snapshot will lose encrypted head when source volume is encrypted in RBD''. Assigned to haixin. - https://bugs.launchpad.net/cinder/+bug/1922013 '' IBM SVF driver: GMCV Add vols to a group fails even if rcrel and rccg are in the same state". Unassigned. Low: - https://bugs.launchpad.net/cinder/+bug/1922255 "Dell PowerVault PVMEISCSIDriver driver cannot manage volumes". Unassigned. - https://bugs.launchpad.net/python-cinderclient/+bug/1922749 "Top-level client doesn't support auth v3". Unassigned. Undecided: - https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1921897 "fast-diff in default feature set breaks 'rbd map'". Assigned to Eric Harney. Regards, Sofi -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[keystone]improvments in mapping models/support for JWT tokens
by pregusia 07 Apr '21

07 Apr '21

Hello members! Please direct your attention to some keystone modyfications - to be more precisly two of them: (1) extension to mapping engine in order to support multiple projects and assigning project by id (2) extension to authorization mechanisms - add JWT token support Ad (1): Currently mapping engine can map multiple projects for user (as stated in https://docs.openstack.org/keystone/pike/advanced-topics/federation/mapping…) but this support lacks from (a) dynamic mapping projects from assertion (for ex. projects ids are in assertion) and (b) mapping engine cannot map to list from assertion (if in asertion we have "some_field": [ id1, id2, ..., idN ] - then using this field by substitution - eg "{1}" wont map this to whole needed structure of projects in mapping). In patch I extend mapping schema by adding new field "projects_spec" of type string. This field contains string formated like project1ID:role1:role2:..:roleN,project2ID:role1:...:roleN,... and this is mapped into proper structure { "projects": [ { "id": .., "roles": [ ... ] }, ... ] } this allows the identity provider to supply permission-like informations about what projects user should have access to. The implementation is not ideal, and some work shoud be do to make it more user-friendly (for ex. configuration options for auto-creating projects by ID, and options for allowing user to log-in if project-by-id not exists etc) This is only proof of concept and question if this direction is proper according to keystone development. Ad (2) This patch adds new authorization protocol - jwt_token. It allows to access endpoint '/v3/OS-FEDERATION/identity_providers/{IDP_NAME}/protocols/jwt_token/auth' (and obtain keystone auth token) using JWT token in Authorization header field. Header value is checked against proper formating and JWT token is extracted from it. Then, token is validated against public key (RS256/RS512 algorithms only supported), against issuer and expiration date and others. Then, if this end with success, payload from token is supplied to mapping engine, when new user (and according to (1), it's permission to projects for ex.) is created and given access to proper projects/roles. Happy reviewing! pregusia

2 2

[cinder] Review of tiny patch to add Ceph RBD fast-diff to cinder-backup
by Christian Rohmann 07 Apr '21

07 Apr '21

Hey everyone, I wrote a tiny patch to add the Ceph RDB feature of fast-diff to backups created by cinder-backup: * https://review.opendev.org/c/openstack/cinder/+/766856/ Could someone please take a peek and let me know of this is sufficient to be merged? Thanks and with kind regards, Christian

3 12

[victoria]oslo_privsep.daemon.FailedToDropPrivileges
by 朝阳未烈 07 Apr '21

07 Apr '21

Hi, everyone: I tried to build an instance on the compute node but failed. I am sure that every necessary connection has been built. And I found the same error information on the controller node and the compute node , in /var/log/neutron/linuxbride-agent.log That is information: INFO neutron.common.config [-] Logging enabled! 2021-04-07 11:30:52.866 2182 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 17.1.0 2021-04-07 11:30:52.867 2182 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'ens160'} 2021-04-07 11:30:52.867 2182 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {} 2021-04-07 11:30:52.868 2182 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/linuxbridge_agent.ini', '--config-dir', '/etc/neutron/conf.d/neutron-linuxbridge-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpm5d0ytiv/privsep.sock'] 2021-04-07 11:30:53.346 2182 CRITICAL oslo.privsep.daemon [-] privsep helper command exited non-zero (1) 2021-04-07 11:30:53.346 2182 CRITICAL neutron [-] Unhandled error: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1) 2021-04-07 11:30:53.346 2182 ERROR neutron Traceback (most recent call last): 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/bin/neutron-linuxbridge-agent", line 10, in <module> 2021-04-07 11:30:53.346 2182 ERROR neutron     sys.exit(main()) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/cmd/eventlet/plugins/linuxbridge_neutron_agent.py", line 28, in main 2021-04-07 11:30:53.346 2182 ERROR neutron     agent_main.main() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1052, in main 2021-04-07 11:30:53.346 2182 ERROR neutron     manager = LinuxBridgeManager(bridge_mappings, interface_mappings) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 79, in __init__ 2021-04-07 11:30:53.346 2182 ERROR neutron     self.validate_interface_mappings() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 94, in validate_interface_mappings 2021-04-07 11:30:53.346 2182 ERROR neutron     if not ip_lib.device_exists(interface): 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 748, in device_exists 2021-04-07 11:30:53.346 2182 ERROR neutron     return IPDevice(device_name, namespace=namespace).exists() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 328, in exists 2021-04-07 11:30:53.346 2182 ERROR neutron     return privileged.interface_exists(self.name, self.namespace) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 246, in _wrap 2021-04-07 11:30:53.346 2182 ERROR neutron     self.start() 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 258, in start 2021-04-07 11:30:53.346 2182 ERROR neutron     channel = daemon.RootwrapClientChannel(context=self) 2021-04-07 11:30:53.346 2182 ERROR neutron   File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 367, in __init__ 2021-04-07 11:30:53.346 2182 ERROR neutron     raise FailedToDropPrivileges(msg) 2021-04-07 11:30:53.346 2182 ERROR neutron oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1) 2021-04-07 11:30:53.346 2182 ERROR neutron     And it is the configuration in /etc/sudoer.d/neutron below:   Defaults:neutron !requiretty neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf * neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf     I googled for the solution but they didn’t matter. How can I solve this problem? Thanks for your advicement!

2 1

[docs] Project guides in PDF format
by Peter Matulis 07 Apr '21

07 Apr '21

Hello, I understand that there was a push [1] to make PDFs available for download from the published pages on docs.openstack.org for the various guides (Sphinx projects). Is there any steam left in this initiative? The next best thing is to have something at the repository level that people can work with. Currently, there are a limited number of guides that can be converted to PDF from the openstack-manuals repository. What is the suggested implementation for individual projects? Thanks, Peter Matulis [1]: https://etherpad.opendev.org/p/train-pdf-support-goal

5 11

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 06 Apr '21

06 Apr '21

Following the suggestion from https://docs.openstack.org/swift/latest/overview_auth.html I set 'log_level=debug' for authtoken. Now I'm seeing more errors in the glance-api logs: Apr 06 15:07:40 infra1-glance-container-99614ac2 glance-wsgi-api[1837]: 2021-04-06 15:07:38.197 1837 WARNING keystonemiddleware.auth_token [req-ad8d0db9-b630-4237-9fff-d7ff282155d2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Identity response: {"error": {"code": 500, "title": "Internal Server Error", "message": "An unexpected error prevented the server from fulfilling your request."}}: keystoneauth1.exceptions.http.InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-55767c13-a450-4f81-90ab-5a7af6b3f672) Apr 06 15:07:46 infra1-glance-container-99614ac2 uwsgi[1822]: DAMN ! worker 13 (pid: 1836) died, killed by signal 9 :( trying respawn ... Here's the full log: http://paste.openstack.org/show/804208/ The options you suggested @Dmitriy are still there in user_variables.yml: glance_glance_api_conf_overrides: keystone_authtoken: service_token_roles_required: True service_token_roles: service Kind regards, Oliver

2 1

[dev][nova] Problem about vm migration compatibility check
by Guoyi Tu 06 Apr '21

06 Apr '21

hi there, In my test environment, i created a vm and configured the cpu with host-model, when I migrate the vm to another host with the same cpu, it failed the migration compatibility check which complains the cpu definition of domain is incompatible with target host cpu. As we know, when the domain configured as above starts, the host-model cpu definition will automatically converted to custom cpu model and with some addtional features that the KVM supported, these addtional features may contains features that the host doesn't support. In the code, the compatibility of the target host is check by calling compareCPU()(libvirt API). The compareCPU() can only recongnize the features probed by cpuid instruction on the host, but it may not recognize the features of cpu definition of domain xml (virsh dumpxml domainname) when the domain running. So the compatibility check will fail when KVM support one or more features which is considerd as disabled by the cpuid instuction. I think we should call compareHypervisorCPU() or something like that (supported by libvirt since v4.4.0) instead of compareCPU() to check the migration compatibility. My test environment is as follow: host cpu: Cascadelake libvirt-6.9 qemu-5.0 host-model cpu: <mode name='host-model' supported='yes'> <model fallback='forbid'>Cascadelake-Server</model> <vendor>Intel</vendor> <feature policy='require' name='ss'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='umip'/> <feature policy='require' name='pku'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='stibp'/> <feature policy='require' name='arch-capabilities'/> <feature policy='require' name='xsaves'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='rdctl-no'/> <feature policy='require' name='ibrs-all'/> <feature policy='require' name='skip-l1dfl-vmentry'/> <feature policy='require' name='mds-no'/> <feature policy='require' name='pschange-mc-no'/> <feature policy='disable' name='hle'/> <feature policy='disable' name='rtm'/> </mode> The hypervisor, umip, pschange-mc-no features block the compatibility check -- Best Regards, Guoyi Tu

2 1

How to modify staging-ovirt parameters
by Gianluca Cecchi 06 Apr '21

06 Apr '21

Hello, I have a test lab with Queens on CentOS 7 deployed with TripleO, that I use sometimes to mimic an OSP 13 environment. The Openstack nodes are oVirt VMS. I moved these VMs from one oVirt infrastructure to another one. The Openstack environment starts and operates without problems (3 controllers, 2 computes and 3 cephs), but I don't know how to modify the staging-ovirt parameters. At oVirt side I re-created as before a user ostackpm@internal with sufficient power and I also set the same password. But the ip address of the engine cannot be the same. When I deployed the environment I put all the parameters inside the instackenv.json file to do introspection and deploy. Where are they put on undercloud? Is there a way to update them? Perhaps some database entries or other kind of repos where I can update the ip? Thanks in advance, Gianluca

2 2

Unable to retrieve ceph docker image for Queens
by Gianluca Cecchi 06 Apr '21

06 Apr '21

Hello, I'm trying to extend an old small test lab Queens environment with dedicated Ceph Storage nodes. I would like to add some disks to the storage nodes and so I customize my original templates/ceph-config.yaml adding them under devices section: parameter_defaults: CephAnsibleDisksConfig: osd_scenario: lvm osd_objectstore: bluestore devices: - /dev/sda - /dev/sdb - /dev/sdc . . . When I run the overcloud deploy I get this in ceph-install-workflow.log 2021-04-06 21:24:57,955 p=32618 u=mistral | TASK [ceph-docker-common : pulling docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 image] *** 2021-04-06 21:24:57,955 p=32618 u=mistral | Tuesday 06 April 2021 21:24:57 +0200 (0:00:00.153) 0:06:17.978 ********* 2021-04-06 21:25:13,206 p=32618 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 image (3 retries left). . . . 2021-04-06 21:26:03,715 p=32618 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 image (1 retries left). 2021-04-06 21:26:28,839 p=32618 u=mistral | fatal: [172.23.0.232]: FAILED! => {"attempts": 3, "changed": false, "cmd": ["timeout", "300s", "docker", "pull", "docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64"], "delta": "0:00:15.022274", "end": "2021-04-06 21:25:48.701980", "msg": "non-zero return code", "rc": 1, "start": "2021-04-06 21:25:33.679706", "stderr": "Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)", "stderr_lines": ["Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"], "stdout": "Trying to pull repository docker.io/ceph/daemon ... ", "stdout_lines": ["Trying to pull repository docker.io/ceph/daemon ... "]} So it seems the docker images are not there any more. My existing ceph nodes are using that image: [heat-admin@ostack-ceph0 ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7ad4aeb3f4cb docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 "/entrypoint.sh" 3 weeks ago Up 3 weeks ceph-osd-0 0dc9a9889283 docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 "/entrypoint.sh" 3 weeks ago Up 3 weeks ceph-osd-3 8d543016cce7 docker.io/tripleoqueens/centos-binary-cron:current-tripleo "dumb-init --singl..." 11 months ago Up 3 weeks logrotate_crond [heat-admin@ostack-ceph0 ~]$ Any way or suggestion to manage this? Thanks, Gianluca

1 0

[qa][gate][stable] stable stein|train py2 devstack jobs are broken
by Ghanshyam Mann 06 Apr '21

06 Apr '21

Hello Everyone, During fixing the grenade issue on stable/train[1], there is another issue that came up for py2 devstack based jobs. I have logged the bug on devstack side as of now - https://bugs.launchpad.net/devstack/+bug/1922736 Let's wait for this to fix before you recheck on failing patches. [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021597.ht… -gmann

3 2

[all] Octavia LoadBalancer Error
by Adekunbi Adewojo 06 Apr '21

06 Apr '21

Hi there, I recently deployed a load balancer on our openstack private cloud. I used this manual - https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html to create the load balancer. However, after creating and trying to access it, it returns an error message saying "No server is available to handle this request". Also on the dashboard, "Operating status" shows offline but "provisioning status" shows active. I have two web applications as members of the load balancer and I can individually access those web applications. Could someone please point me in the right direction? Thanks.

3 3

all Octavia LoadBalancer
by Adekunbi Adewojo 06 Apr '21

06 Apr '21

Hi there, I recently deployed a load balancer on our openstack private cloud. I used this manual - https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html to create the load balancer. However, after creating and trying to access it, it returns an error message saying "No server is available to handle this request". Also on the dashboard, "Operating status" shows offline but "provisioning status" shows active. I have two web applications as members of the load balancer and I can individually access those web applications. Could someone please point me in the right direction? Thanks.

3 3

[tripleo] master and victoria promotions delayed
by Wesley Hayutin 06 Apr '21

06 Apr '21

0/ Hey just want to communicate that promotions for master and victoria are being delayed due to promotion-blockers. If you have a merged patch in a repo that is not gated by tripleo jobs you will NOT be able to use that patch until we promote. http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?org… Please review the following bugs to better understand what is blocking. https://bugs.launchpad.net/tripleo/+bugs?field.tag=promotion-blocker&orderb… I'll note, any tempest test results logged in bugs are most likely already skipped and not blocking upstream via https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/… Your focus should be on deployment failures, not tempest failures at this time. Thank you!

1 0

[qa][hacking] Proposing new core reviewers
by Martin Kopec 06 Apr '21

06 Apr '21

Hi all, I'd like to propose Sorin Sbarnea (IRC: zbr) and Radosław Piliszek (IRC: yoctozepto) to hacking core. They both are doing a great upstream work among multiple different projects and volunteered to help us with maintenance of hacking project as well. You can vote/feedback in this email thread. If no objection by 6th of April, we will add them to the list. Regards, -- Martin Kopec

3 3

[neutron] bug deputy report (3/29-4/4)
by Akihiro Motoki 06 Apr '21

06 Apr '21

Hi, I was a bug deputy last week and here is a report. Last week is relatively quite. Please check my report. ## Unassigned * OpenStack Metadata API and OVN in Neutron https://bugs.launchpad.net/neutron/+bug/1921809 It would be nice if OVN folks follow it more while haleyb replied. ## Medium, Assigned * [LB] Linux Bridge iptables firewall does not work without "ipset" https://bugs.launchpad.net/neutron/+bug/1922127 Assigned to ralonsoh ## Fix Released * Strings in tags field is limited to 60 chars https://bugs.launchpad.net/neutron/+bug/1921713 ## Almost RFE * allow using tap device on netdev enabled host https://bugs.launchpad.net/neutron/+bug/1922222 I requested the bug author to provide more information on the background ## RFE * [QoS] Add minimum guaranteed packet rate QoS rule https://bugs.launchpad.net/neutron/+bug/1922237 * [RFE] BFD for BGP Dynamic Routing https://bugs.launchpad.net/neutron/+bug/1922716

1 0

[cinder][horizon] Ussuri: Horizon shows error for managed volume(s)
by Eugen Block 06 Apr '21

06 Apr '21

Hi *, I'm struggling with a dashboard error that seems to be wrong, I can't seem to find any existing reports but I could have simply missed it so please point me to any existing reports about that. Anyway, a user imported two volumes from Ceph via 'cinder manage' in our Train version of OpenStack, we upgraded last week to Ussuri. One of those two volumes is bootable and the instance is up and running with both volumes (extended volume group). When clicking on the instance details the Horizon dashboard shows a red window saying "Failed to get attached volume." The dashboard log says: [Tue Apr 06 09:29:56.197238 2021] [wsgi:error] [pid 10140] [remote IP:59482] WARNING horizon.exceptions Recoverable error: volume_image_metadata But I can see both volumes in the details page, so the message seems incorrect. I searched the logs for this error message from before the upgrade but couldn't find any. So this seems to be new in Ussuri, I assume. Does anyone have the same experience with managed volumes? Thanks and best regards, Eugen

1 0

[victoria][neutron][horizon]l3-agent+port-forwarding
by Luke Camilleri 06 Apr '21

06 Apr '21

Hello everyone, I have enable the L3 extension for port-forwarding and can succesfully port-forward traffic after assigning an additional floating IP to the project. I would like to know if it is possible to enable the corresponding horizon functionality for this extension (port-forwarding) please? Regards

3 4

RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 06 Apr '21

06 Apr '21

Sorry that was a typo. Stopping RMQ during the removal of the *second* controller is what causes the problem. Is there a way to tell Centos 8 Train to use RMQ 3.7.24 instead of 3.7.28? -----Original Message----- From: Braden, Albert Sent: Thursday, April 1, 2021 9:34 AM To: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller I did some experimenting and it looks like stopping RMQ during the removal of the first controller is what causes the problem. After deploying the first controller, stopping the RMQ container on any controller including the new centos8 controller will cause the entire cluster to stop. This crash dump appears on the controllers that stopped in sympathy: https://paste.ubuntu.com/p/ZDgFgKtQTB/ This appears in the RMQ log: https://paste.ubuntu.com/p/5D2Qjv3H8c/ -----Original Message----- From: Braden, Albert Sent: Wednesday, March 31, 2021 8:31 AM To: openstack-discuss(a)lists.openstack.org Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller Centos7: {rabbit,"RabbitMQ","3.7.24"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, Centos8: {rabbit,"RabbitMQ","3.7.28"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, When I deploy the first Centos8 controller, RMQ comes up with all 3 nodes active and seems to be working fine until I shut down the 2nd controller. The only hint of trouble when I replace the 1st node is this error message the first time I run the deployment: https://paste.ubuntu.com/p/h9HWdfwmrK/ and the crash dump that appears on control2: crash dump log: https://paste.ubuntu.com/p/MpZ8SwTJ2T/ First 1500 lines of the dump: https://paste.ubuntu.com/p/xkCyp2B8j8/ If I wait for a few minutes then RMQ recovers on control2 and the 2nd run of the deployment seems to work, and there is no trouble until I shut down control1. -----Original Message----- From: Mark Goddard <mark(a)stackhpc.com> Sent: Wednesday, March 31, 2021 4:14 AM To: Braden, Albert <C-Albert.Braden(a)charter.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Tue, 30 Mar 2021 at 13:41, Braden, Albert <C-Albert.Braden(a)charter.com> wrote: > > I’ve created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: > > > > https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… > > > > I used the instructions here to successfully remove and replace control0 with a Centos8 box > > > > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > > > > After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, > > {'rabbit@chrnc-void-testupgrade-control-1',[]}, > > {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] > > > > After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: > > > > kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 > > … > > control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 > > > > After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. > > > > If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', > > 'rabbit@chrnc-void-testupgrade-control-0-replace']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, > > {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] > > > > But my hypervisors are down: > > > > (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | > > | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | > > | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > > > When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: > > > > 172.16.2.31 compute0 > > 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. > > 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > > > In the RMQ logs I see this every 10 seconds: > > > > 172.16.1.132 control2 > > [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 > > 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > client unexpectedly closed TCP connection > > 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) > > 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e > > 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' > > 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > > > Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? Hi Albert, Could you share the versions of RabbitMQ and erlang in both versions of the container? When initially testing this setup, I think we had 3.7.24 on both sides. Perhaps the CentOS 8 version has moved on sufficiently to become incompatible? Mark > > > > I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

2 1

[docs][release] Creating Xena's landing pages
by Herve Beraud 06 Apr '21

06 Apr '21

Hello Docs team, This is a friendly reminder from the release team, I think that it should be safe for you to apply your process to create the new release series landing pages for docs.openstack.org. All stable branches are now created. If you want you can do the work before the final release date to avoid having to synchronize with the release team on that day. Let us know if you have any questions. Cheers -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 1

[all] Gate resources and performance
by Dan Smith 06 Apr '21

06 Apr '21

Hi all, I have become increasingly concerned with CI performance lately, and have been raising those concerns with various people. Most specifically, I'm worried about our turnaround time or "time to get a result", which has been creeping up lately. Right after the beginning of the year, we had a really bad week where the turnaround time was well over 24 hours. That means if you submit a patch on Tuesday afternoon, you might not get a test result until Thursday. That is, IMHO, a real problem and massively hurts our ability to quickly merge priority fixes as well as just general velocity and morale. If people won't review my code until they see a +1 from Zuul, and that is two days after I submitted it, that's bad. Things have gotten a little better since that week, due in part to getting past a rush of new year submissions (we think) and also due to some job trimming in various places (thanks Neutron!). However, things are still not great. Being in almost the last timezone of the day, the queue is usually so full when I wake up that it's quite often I don't get to see a result before I stop working that day. I would like to ask that projects review their jobs for places where they can cut out redundancy, as well as turn their eyes towards optimizations that can be made. I've been looking at both Nova and Glance jobs and have found some things I think we can do less of. I also wanted to get an idea of who is "using too much" in the way of resources, so I've been working on trying to characterize the weight of the jobs we run for a project, based on the number of worker nodes required to run all the jobs, as well as the wall clock time of how long we tie those up. The results are interesting, I think, and may help us to identify where we see some gains. The idea here is to figure out[1] how many "node hours" it takes to run all the normal jobs on a Nova patch compared to, say, a Neutron one. If the jobs were totally serialized, this is the number of hours a single computer (of the size of a CI worker) would take to do all that work. If the number is 24 hours, that means a single computer could only check *one* patch in a day, running around the clock. I chose the top five projects in terms of usage[2] to report here, as they represent 70% of the total amount of resources consumed. The next five only add up to 13%, so the "top five" seems like a good target group. Here are the results, in order of total consumption: Project % of total Node Hours Nodes ------------------------------------------ 1. TripleO 38% 31 hours 20 2. Neutron 13% 38 hours 32 3. Nova 9% 21 hours 25 4. Kolla 5% 12 hours 18 5. OSA 5% 22 hours 17 What that means is that a single computer (of the size of a CI worker) couldn't even process the jobs required to run on a single patch for Neutron or TripleO in a 24-hour period. Now, we have lots of workers in the gate, of course, but there is also other potential overhead involved in that parallelism, like waiting for nodes to be available for dependent jobs. And of course, we'd like to be able to check more than patch per day. Most projects have smaller gate job sets than check, but assuming they are equivalent, a Neutron patch from submission to commit would undergo 76 hours of testing, not including revisions and not including rechecks. That's an enormous amount of time and resource for a single patch! Now, obviously nobody wants to run fewer tests on patches before they land, and I'm not really suggesting that we take that approach necessarily. However, I think there are probably a lot of places that we can cut down the amount of *work* we do. Some ways to do this are: 1. Evaluate whether or not you need to run all of tempest on two configurations of a devstack on each patch. Maybe having a stripped-down tempest (like just smoke) to run on unique configs, or even specific tests. 2. Revisit your "irrelevant_files" lists to see where you might be able to avoid running heavy jobs on patches that only touch something small. 3. Consider moving some jobs to the experimental queue and run them on-demand for patches that touch particular subsystems or affect particular configurations. 4. Consider some periodic testing for things that maybe don't need to run on every single patch. 5. Re-examine tests that take a long time to run to see if something can be done to make them more efficient. 6. Consider performance improvements in the actual server projects, which also benefits the users. If you're a project that is not in the top ten then your job configuration probably doesn't matter that much, since your usage is dwarfed by the heavy projects. If the heavy projects would consider making changes to decrease their workload, even small gains have the ability to multiply into noticeable improvement. The higher you are on the above list, the more impact a small change will have on the overall picture. Also, thanks to Neutron and TripleO, both of which have already addressed this in some respect, and have other changes on the horizon. Thanks for listening! --Dan 1: https://gist.github.com/kk7ds/5edbfacb2a341bb18df8f8f32d01b37c 2; http://paste.openstack.org/show/C4pwUpdgwUDrpW6V6vnC/

12 36

[keystone][horizon][Victoria] scope-based policy problems
by Adam Tomas 06 Apr '21

06 Apr '21

Hi, I have some questions about horizon and keystone policies : Im trying to achieve "domain_admin" role with the ability to add/remove/update projects in particular domain and add/update/remove users in the same domain (and of course be able to see instances, networks, etc. in this domain). As described here http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021105.ht… <http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021105.ht…> : "Horizon does not support the system-scoped token yet as of Victoria and coming Wallaby.” So there is no way to write json/not scope-based policy for horizon and scope-based policy for keystone, because it will not work due to lack of scope information in horizon’s token? So the question is how the policies should look like? Is it possible at all to achieve such „domain admin” role? How in different way allow one user to add/remove/update projects and add/update/remove users? Another thing is, that if I use something like this in horizon/keystone policy: "identity:list_users_in_group": "rule:admin_required or (role:domain_admin and domain_id:%(domain_id)s)” then (besides of that domain users) there is also admin account in the list (so I assume admin „belongs” to all domains) - how to prevent newly created domain_admin from seeing admin account and making changes to that account? It really holds up my whole project, can you help mi guys? Best regards Adam

1 0

[kolla][glance] Few questions about images.
by Eddie Yen 06 Apr '21

06 Apr '21

Hi everyone, I want to ask about the image operating permission & Windows images issue we met since we can't find any answers on the internet. 1. Now we're still using Rocky with Ceph as storage. Sometimes we need to re-pack the image on the Openstack. We used to save as snapshot (re-pack by Nova ephemeral VM) or upload to image (re-pack by volume), then set snapshot/image's visibility to the public. But since Rocky, we can't do this anymore because when we try to set public, Horizon always shows "not enough permission" error. The workaround we're using for now is creating a nova snapshot after re-pack, download the snapshot, then upload the snapshot again as public. But it's utterly wasting time if the images are huge. So we want to know how to unleash this limitation can let us just change snapshot to public at least. 2. Openstack uses virtio as a network device by default, so we always install a virtio driver when packing Windows images. As the network performance issue in GSO/TSO enablement, we also need to disable them in device properties. But since Windows 10 2004 build (my thought) device properties always reset these settings after Sysprep. We found there's a workaround [1] to solve this issue, but may not work sometimes. Is there a better way to solve this issue? Many thanks, Eddie. [1] PersistAllDeviceInstalls | Microsoft Docs <https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/unatten…>

1 1