February 2021 - openstack-discuss

[heat] [kolla] Stack creation fails
by Braden, Albert 24 Feb '21

24 Feb '21

I'm using a heat template to try and build an Openstack cluster on VMs and it is failing in 2 different ways. It starts out fine for both: (openstack) [abraden@chrnc-dev-os-build-01 kolla_multinode]$ os stack create -t kolla_multinode.yaml upgrade-stack +---------------------+----------------------------------------------------------------------------------------+ | Field | Value | +---------------------+----------------------------------------------------------------------------------------+ | id | 7b4a49d2-0418-453a-abde-5e510b588b71 | | stack_name | upgrade-stack | | description | Deploy virtualized multi-node OpenStack and Ceph using kolla-ansible and ceph-ansible. | | | | | creation_time | 2021-02-23T21:17:02Z | | updated_time | None | | stack_status | CREATE_IN_PROGRESS | | stack_status_reason | Stack CREATE started | +---------------------+----------------------------------------------------------------------------------------+ Sometimes it stays in CREATE_IN_PROGRESS status for a long time, and when I look at volumes I see one or more volumes stuck in "creating" status [1]. If I set the stuck volumes to "error" status then heat retries and the cluster builds successfully. Other times the keypair fails. First it goes to DELETE_COMPLETE status and then to CREATE_FAILED [2], and the the stack goes to CREATE_FAILED status. When this happens no volumes are created, and I haven't found a workaround for this failure. Where should I be looking for the issue? Nothing in the log is jumping out at me. RMQ and keystone seem fine. Searching for the id of the failed stack or the stuck volume in /var/log/kolla/* doesn't find any errors or warnings. [1] volumes<https://paste.ubuntu.com/p/ZBhn7TQ2Hv/> [2] failed key<https://paste.ubuntu.com/p/gGwDRyT2Nd/> I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

1 1

[security][security sig] No meeting tomorrow
by Gage Hugo 24 Feb '21

24 Feb '21

Hello everyone, The security SIG meeting for tomorrow is cancelled, we plan on meeting next week at the usual time. Thanks

1 0

[victoria][neutron] qos-fip extension cannot be enabled with OVN
by Malik Obaid 24 Feb '21

24 Feb '21

Hi, I am using Openstack Victoria release on Ubuntu 20.04. While configuring QOS policy I am facing 2 issues. 1. The command 'openstack extension list' does not show qos-fip, despite qos-fip having been added to the list of extension_drivers. 2. An error is raised when trying to assign a qos-policy to a FIP: BadRequestException: 400: Client Error for url: http://172.16.30.46:9696/v2.0/floatingips/774dd653-3a9b-40d7-ade6-7cf5a3a4b…, Unrecognized attribute(s) 'qos_policy_id' I would really appreciate any help in this regard. Thank you. Regards, Malik Obaid

2 1

[magnum][ingress-nginx][cert-manager] issuing letsencrypt certs
by Ionut Biru 24 Feb '21

24 Feb '21

Hi guys, Not sure if we need some new policy on deploying kubernetes 1.18+ but I'm kinda stuck and I don't know what else to do. I'm trying in the past days to generate letsencrypt ssl for a simple nginx deployment I'm deploying Kubernetes 1.19.8 or 1.20.4 cluster on Openstack using Magnum without any ingress controller. I'm deploying ingress-nginx 0.44 using: wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44… Cert-manager 1.2.0 wget https://github.com/jetstack/cert-manager/releases/download/v1.2.0/cert-mana… I'm creating a nginx deployment and service with 2 replicas https://paste.xinu.at/PWu/ Configure ingress for my host: https://paste.xinu.at/7c7FH/ Configure the issuer: https://paste.xinu.at/Bf6/ Reconfigure ingress: https://paste.xinu.at/o1j5wD/ kubectl apply -f deploy.yaml kubectl apply -f cert-manager.yaml kubectl apply -f nginx-deployment.yaml kubectl apply -f ioni_ingress.yaml kubectl apply -f prod_issuer.yaml Error in events: Error presenting challenge: pods "cm-acme-http-solver-" is forbidden: PodSecurityPolicy: unable to admit pod: [] Error in kubernetes logs(ssh on node) https://paste.xinu.at/9aMJ/ -- Ionut Biru - https://fleio.com

1 0

[ops] Where is the instance Action Log stored and how to externally access it and send to ELK?
by Linebarger, John 23 Feb '21

23 Feb '21

Where is the instance Action Log stored (the one you can see on the Horizon dashboard if you select an instance) and how can it be externally accessed, either by CLI or HTTPS or in any other way? It doesn't seem to map to anything stored in the Nova logs in /var/log/nova, at least that I can see. We want to send instance Action Logs to an ELK stack for indexing and display, but are not yet able to figure out how to do so. John M. Linebarger, PhD, MBA Principal Member of Technical Staff Sandia National Laboratories (Work) 505-845-8282

2 1

[all][qa] Gate status of stable stein's legacy jobs with tempest plugins
by Ghanshyam Mann 23 Feb '21

23 Feb '21

Hello Everyone, We recently pinned the Tempest version 26.0.0 in devstack for stable/stein branch testing.[1]. In CI/CD workflow tempest venv can be re-created many times in devstack, devstack-gate (for legacy jobs) or run-tempest roles etc. With the Tempest 26.0.0 version, we need to make tempest venv to use the stable constraints instead of master constraints. And used constraints should be consistent otherwise tempest venv gets recreated which end up removing the previously installed tempest plugins. We have taken care of constraints consistency for zuulv3 native jobs in run-tempest ansible role but for the legacy job which uses the devstack-gate path still use the master constraints and recreate the tempest venv. So your legacy job which uses the tempest plugins will be failing. Example: designate-devstack-pdns4 - https://zuul.opendev.org/t/openstack/build/35fd0a4c76e0482e88ed32ef0840291d I have pushed the fixes which are on devstack, devstack-gate, and tempest(tempest patch is merged). If you are seeing a similar issue please do not recheck and wait for the below patches to merge: - https://review.opendev.org/c/openstack/devstack-gate/+/776722 - https://review.opendev.org/c/openstack/devstack/+/776811 [1] https://review.opendev.org/c/openstack/devstack/+/774103 -gmann

1 1

[glance] Replaced parameters "default_store" and "glance_store" breaks cooperation between cinder and glance in rocky release
by Jan Wasilewski 23 Feb '21

23 Feb '21

Hello, I am writing to you, because we faced an issue with broken cooperation between cinder and glance. In our logs(glance-api) we observed such warning message: 2021-02-22 15:30:02.894 756 WARNING oslo_config.cfg [-] Deprecated: Option "stores" from group "glance_store" is deprecated for removal ( This option is deprecated against new config option ``enabled_backends`` which helps to configure multiple backend stores of different schemes. This option is scheduled for removal in the Stein development cycle. ). Its value may be silently ignored in the future. 2021-02-22 15:30:02.896 756 WARNING oslo_config.cfg [-] Deprecated: Option "default_store" from group "glance_store" is deprecated for removal ( This option is deprecated against new config option ``default_backend`` which acts similar to ``default_store`` config option. This option is scheduled for removal in the Stein development cycle. ). Its value may be silently ignored in the future. And we simply wanted to solve this problem, by changing option glance_store to enabled_backends and default_store to default_backend: @@ -2079,7 +2079,7 @@ pool_timeout = 60 # cycle. #stores = file,http -stores = glance.store.rbd.Store +enabled_backends = glance.store.rbd.Store # DEPRECATED: # The default scheme to use for storing images. @@ -2134,7 +2134,7 @@ stores = glance.store.rbd.Store # cycle. #default_store = file -default_store = rbd +default_backend = rbd But then we realized that we're not able to create volumes from new images, because we hit an issue: 2021-02-22 22:45:08.805 5910 ERROR cinder.scheduler.filter_scheduler [req-fe8e8aa2-dd8a-470b-9d70-dbdc757b48d9 54df39e1b74149bcb8cf9ad3eb957d48 afdfb965515f4724a21b6823764ec36d - default default] Error scheduling 9de37fe8-4f9e-4c29-8cfa-c1a71e8c68b7 from last vol-service: cinder-05@huawei_backend#StoragePool001 : [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task\n result = task.execute(**arguments)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 1059, in execute\n **volume_spec)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 818, in _wrapper\n return r.call(f, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n', u' File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n', u' File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 959, in _create_from_image\n image_service)\n', u' File "<decorator-gen-238>", line 2, in _prepare_image_cache_entry\n', u' File "/usr/lib/python2.7/dist-packages/cinder/coordination.py", line 151, in _synchronized\n return f(*a, **k)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 757, in _prepare_image_cache_entry\n update_cache=True)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 831, in _create_from_image_cache_or_download\n backend_name) as tmp_image:\n', u' File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__\n return self.gen.next()\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 832, in fetch\n fetch_verify_image(context, image_service, image_id, tmp)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 451, in fetch_verify_image\n None, None)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 377, in fetch\n tpool.Proxy(image_file))\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 358, in download\n _reraise_translated_image_exception(image_id)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 602, in _reraise_translated_image_exception\n six.reraise(type(new_exc), new_exc, exc_trace)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 356, in download\n image_chunks = self._client.call(context, \'data\', image_id)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 225, in call\n return getattr(controller, method)(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/common/utils.py", line 545, in inner\n return RequestIdProxy(wrapped(*args, **kwargs))\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 208, in data\n resp, body = self.http_client.get(url)\n', u' File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 328, in get\n return self.request(url, \'GET\', **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 349, in request\n return self._handle_response(resp)\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 98, in _handle_response\n raise exc.from_response(resp, resp.content)\n', u'HTTPInternalServerError: HTTPInternalServerError (HTTP 500)\n'] Which is visible in cinder-scheduler log file. When we reverted a change, everything started to operate normally. I'm just wondering how we can skip this warning message and makes cinder and glance to cooperate correctly, especially that we're planning to make an upgrade from Rocky to Stein soon and this option is planned to be removed in the Stein release. Thanks in advance for your help here. Pozdrawiam, Jan Wasilewski Administrator systemów i sieci CI ATM Tel: +48 691 493 327 jan.wasilewski(a)atman.pl<mailto:jan.wasilewski@atman.pl>, www.atman.eu<http://www.atman.eu/> Informacja o zasadach przetwarzania danych osobowych przez ATM S.A. znajduje się pod tym linkiem<http://www.atman.pl/Informacja-o-przetwarzaniu-danych-osobowych-cabout-pol-…>.

2 1

[tripleo] migrating master from CentOS-8 to CentOS-8-Stream
by Wesley Hayutin 23 Feb '21

23 Feb '21

Greetings, Just a heads up over the course of the next few weeks upstream TripleO should see a transparent migration from CentOS-8 to CentOS-8-Stream. We do have a few options with regards to how the transition will take place. First and foremost we're going to migrate the master branch only at this time. *Question 1: Artifacts* Option1: New namespaces for artifacts: Containers: https://hub.docker.com/r/tripleomasterstream/ or some combination Images: http://images.rdoproject.org/centosstream8 or other combination Option2: Master content and namespaces is overwritten with centos-8-stream containers and images and will retain the paths and namespaces. Containers: https://hub.docker.com/u/tripleomaster Images: http://images.rdoproject.org/centos8/master/ *Question 2: job names* We can update the master jobs to include "stream" in the name and be explicit for the distro name OR: We can leave the job names as is and just communicate that "centos-8" is now really centos-8-stream Now's your chance to weigh in prior to the upcoming changes. Thanks!

5 5

Unexpected MAC Address Change
by Lazuardi Nasution 23 Feb '21

23 Feb '21

Hi, Today, I find some of our Windows Server VMs cannot be accessed. After some inspection, I find that there is expected MAC address change on those VMs. There are inconsistencies between "ps aux | grep instance-***" output on such compute nodes where affected VMs reside, with "virsh dumpxml instance-***" on the same compute nodes. The "virsh dumpxml instance-***" output is consistent with TAP interface MAC address on "ip link" output and the MAC address of port list on OpenStack dashboard/CLI. Following are one of those outputs. This weirdness is back to normal after I do a live migration to another compute node. virsh dumpxml instance-000003e5 ... <interface type='bridge'> <mac address='fa:16:3e:ef:4e:3b'/> <source bridge='br-int'/> ... ps aux | grep instance-000003e5 ... netdev=hostnet0,id=net0,mac=fa:16:3e:8e:92:f0 ... What can change the MAC address on "ps aux | grep instance-***" output? This weirdness happens on Ussuri with Ubuntu 20.04 and OVN. Is this a bug? What kind of bug? Is there any other information should I look for to find the root cause? Best regards,

1 0

[largescale-sig] Video meeting Feb 24: "Regions vs. Cells"
by Thierry Carrez 23 Feb '21

23 Feb '21

Hi everyone, In order to increase its reach, the Large Scale SIG will organize video meetings around specific scaling topics, just one hour of discussion bootstrapped by a short presentation from operators of large scale deployments of OpenStack. Our first video meeting will be next Wednesday, February 24, at 1500utc on Zoom (link/password will be provided on this thread the day before the event). The theme will be: "Regions vs. Cells" quickstarted by a presentation from Belmiro Moreira (CERN). If you're interested in sharing your experience on this topic, or just want to learn more about that, please mark your calendar and join us next Wednesday ! -- Thierry Carrez

1 1