May 2020 - openstack-discuss - lists.openstack.org

Ussuri RDO Release Announcement
by Amy Marrich 28 May '20

28 May '20

If you're having trouble with the formatting, this release announcement is available online https://blogs.rdoproject.org/2020/05/rdo-ussuri-released/ --- *RDO Ussuri Released* The RDO community is pleased to announce the general availability of the RDO build for OpenStack Ussuri for RPM-based distributions, CentOS Linux and Red Hat Enterprise Linux. RDO is suitable for building private, public, and hybrid clouds. Ussuri is the 21st release from the OpenStack project, which is the work of more than 1,000 contributors from around the world. The release is already available on the CentOS mirror network at http://mirror.centos.org/centos/8/cloud/x86_64/openstack-ussuri/. The RDO community project curates, packages, builds, tests and maintains a complete OpenStack component set for RHEL and CentOS Linux and is a member of the CentOS Cloud Infrastructure SIG. The Cloud Infrastructure SIG focuses on delivering a great user experience for CentOS Linux users looking to build and maintain their own on-premise, public or hybrid clouds. All work on RDO and on the downstream release, Red Hat OpenStack Platform, is 100% open source, with all code changes going upstream first. PLEASE NOTE: At this time, RDO Ussuri provides packages for CentOS8 only. Please use the previous release, Train, for CentOS7 and python 2.7. *Interesting things in the Ussuri release include:* - Within the Ironic project, a bare metal service that is capable of managing and provisioning physical machines in a security-aware and fault-tolerant manner, UEFI and device selection is now available for Software RAID. - The Kolla project, the containerised deployment of OpenStack used to provide production-ready containers and deployment tools for operating OpenStack clouds, streamlined the configuration of external [Ceph]( https://ceph.io/) integration, making it easy to go from Ceph-Ansible-deployed Ceph cluster to enabling it in OpenStack. *Other improvements include:* - Support for IPv6 is available within the Kuryr project, the bridge between container framework networking models and OpenStack networking abstractions. - Other highlights of the broader upstream OpenStack project may be read via https://releases.openstack.org/ussuri/highlights.html. - A new Neutron driver networking-omnipath has been included in RDO distribution which enables the Omni-Path switching fabric in OpenStack cloud. - OVN Neutron driver has been merged in main neutron repositon from networking-ovn. *Contributors* During the Ussuri cycle, we saw the following new RDO contributors: - Amol Kahat - Artom Lifshitz - Bhagyashri Shewale - Brian Haley - Dan Pawlik - Dmitry Tantsur - Dougal Matthews - Eyal - Harald Jensås - Kevin Carter - Lance Albertson - Martin Schuppert - Mathieu Bultel - Matthias Runge - Miguel Garcia - Riccardo Pittau - Sagi Shnaidman - Sandeep Yadav - SurajP - Toure Dunnon Welcome to all of you and Thank You So Much for participating! But we wouldn’t want to overlook anyone. A super massive Thank You to all 54 contributors who participated in producing this release. This list includes commits to rdo-packages and rdo-infra repositories: - Adam Kimball - Alan Bishop - Alan Pevec - Alex Schultz - Alfredo Moralejo - Amol Kahat - Artom Lifshitz - Arx Cruz - Bhagyashri Shewale - Brian Haley - Cédric Jeanneret - Chandan Kumar - Dan Pawlik - David Moreau Simard - Dmitry Tantsur - Dougal Matthews - Emilien Macchi - Eric Harney - Eyal - Fabien Boucher - Gabriele Cerami - Gael Chamoulaud - Giulio Fidente - Harald Jensås - Jakub Libosvar - Javier Peña - Joel Capitao - Jon Schlueter - Kevin Carter - Lance Albertson - Lee Yarwood - Marc Dequènes (Duck) - Marios Andreou - Martin Mágr - Martin Schuppert - Mathieu Bultel - Matthias Runge - Miguel Garcia - Mike Turek - Nicolas Hicher - Rafael Folco - Riccardo Pittau - Ronelle Landy - Sagi Shnaidman - Sandeep Yadav - Soniya Vyas - Sorin Sbarnea - SurajP - Toure Dunnon - Tristan de Cacqueray - Victoria Martinez de la Cruz - Wes Hayutin - Yatin Karel - Zoltan Caplovic *The Next Release Cycle* At the end of one release, focus shifts immediately to the next, Victoria, which has an estimated GA the week of 12-16 October 2020. The full schedule is available at https://releases.openstack.org/victoria/schedule.html. Twice during each release cycle, RDO hosts official Test Days shortly after the first and third milestones; therefore, the upcoming test days are 25-26 June 2020 for Milestone One and 17-18 September 2020 for Milestone Three. *Get Started* There are three ways to get started with RDO. To spin up a proof of concept cloud, quickly, and on limited hardware, try an All-In-One Packstack installation. You can run RDO on a single node to get a feel for how it works. For a production deployment of RDO, use the TripleO Quickstart and you’ll be running a production cloud in short order. Finally, for those that don’t have any hardware or physical resources, there’s the OpenStack Global Passport Program. This is a collaborative effort between OpenStack public cloud providers to let you experience the freedom, performance and interoperability of open source infrastructure. You can quickly and easily gain access to OpenStack infrastructure via trial programs from participating OpenStack public cloud providers around the world. *Get Help* The RDO Project participates in a Q&A service at https://ask.openstack.org. We also have our users(a)lists.rdoproject.org for RDO-specific users and operrators. For more developer-oriented content we recommend joining the dev(a)lists.rdoproject.org mailing list. Remember to post a brief introduction about yourself and your RDO story. The mailing lists archives are all available athttps://mail.rdoproject.org. You can also find extensive documentation on RDOproject.org. The #rdo channel on Freenode IRC is also an excellent place to find and give help. We also welcome comments and requests on the CentOS devel mailing list and the CentOS and TripleO IRC channels (#centos, #centos-devel, and #tripleo on irc.freenode.net) however we have a more focused audience within the RDO venues. *Get Involved* To get involved in the OpenStack RPM packaging effort, check out the RDO contribute pages, peruse the CentOS Cloud SIG page, and inhale the RDO packaging documentation. Join us in #rdo and #tripleo on the Freenode IRC network and follow us on Twitter @RDOCommunity. You can also find us on Facebook and YouTube. Amy Marrich (spotz) https://www.rdoproject.org http://community.redhat.com

1 0

[tc] Monthly update (to become Weekly)
by Mohammed Naser 28 May '20

28 May '20

Hello everyone, Here’s an update of what’s happened in the OpenStack TC during the last month. Moving forward, I’ll be sending an update on a weekly basis. You can get more information by checking for changes in openstack/governance repository. # CHANGES PENDING - Rename ansible-role-lunasa-hsm deliverable: https://review.opendev.org/#/c/731313/ - Add whitebox-tempest-plugin under QA: https://review.opendev.org/#/c/714480/ - Select migrate-to-focal goal for Victoria cycle: https://review.opendev.org/#/c/731213/ - Clarify the support for linux distro: https://review.opendev.org/#/c/727238/ - Add QA branchless projects also in py3.5 support list: https://review.opendev.org/#/c/729325/ - Remove congress and tricircle project teams: https://review.opendev.org/#/c/728818/ - [draft] Add assert:supports-standalone: https://review.opendev.org/#/c/722399/ # PROJECT UPDATES - Adding Storlets in the list of keeping py2: https://review.opendev.org/#/c/720714/ - Transfer Policy team co-lead role: https://review.opendev.org/#/c/722101/ - Add keystoneauth to the python3.5 list: https://review.opendev.org/#/c/721093/ - Add note that OpenStackSDK kept 3.5 support: https://review.opendev.org/#/c/721083/ - Retire i18n-specs: https://review.opendev.org/#/c/721723/ - Migrate from mock to built in unittest.mock: https://review.opendev.org/#/c/722924/ - Migrate from olso.rootwrap to oslo.privsep: https://review.opendev.org/#/c/718177/ - Appoint Yasufumi Ogawa as Tacker PTL: https://review.opendev.org/#/c/719091/ # GOAL UPDATES - Infrastructure project team is now the TaCT SIG: https://review.opendev.org/#/c/721621/ - Add ansible role for managing Luna SA HSM: https://review.opendev.org/#/c/721348/ - Drop deprecated release-management flag for xstatic repos: https://review.opendev.org/#/c/721046/ # ABANDONED CHANGES - Remove CentOS 8 from tested runtimes for Ussuri: https://review.opendev.org/#/c/720005/ - Goals: add container-images: https://review.opendev.org/#/c/720107/ # GENERAL UPDATES - Add goal to migrate CI/CD jobs to Ubuntu Focal: https://review.opendev.org/#/c/728158/ - Fix hacking min version to 3.0.1: https://review.opendev.org/#/c/730161/ - Appoint Javier Peña as rpm-packaging PTL: https://review.opendev.org/#/c/728088/ - Add repository for oslo.metrics: https://review.opendev.org/#/c/725848/ - Require PTL signoff for project-update changes: https://review.opendev.org/#/c/727704/ - Allow for faster addition of projects: https://review.opendev.org/#/c/726932/ - Retire syntribos: https://review.opendev.org/#/c/726507/ - Loosen voting for community goal proposals: https://review.opendev.org/#/c/724142/ - Add TrilioVault Charms to OpenStack Charms: https://review.opendev.org/#/c/720533/ - Cleanup retired repositories: https://review.opendev.org/#/c/724334/ - Move i18n into a SIG: https://review.opendev.org/#/c/721605/ # OTHER - Cap jsonschema 3.2.0 as the minimal version: https://review.opendev.org/#/c/730948/ - Fix bullet list formatting for drop-py2 goal: https://review.opendev.org/#/c/729314/ Thanks and sorry for the delay in getting those out. -- Mohammed Naser VEXXHOST, Inc.

1 0

Re: Uniting TC and UC
by Mohamed Elsakhawy 28 May '20

28 May '20

Hi All, As you may know already, there has been an ongoing discussion to merge UC and TC under a single body. Three options were presented, along with their impact on the bylaws. http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012806… We had several discussions in the UC on the three options as well as the UC items that need to be well-represented under the common committee, and here’s what we propose for the merger: 1- There’s consensus on utilizing option 1, thus requiring no bylaws change and merging AUC and ATC : 1- No bylaws change As bylaws changes take a lot of time and energy, the simplest approach would be to merge the TC and UC without changing the bylaws at all. The single body (called TC) would incorporate the AUC criteria by adding all AUC members as extra-ATC. It would tackle all aspects of our community. To respect the letter of the bylaws, the TC would formally designate 5 of its members to be the 'UC' and those would select a 'UC chair'. But all tasks would be handled together. 2- Two main core UC responsibilities need to continue to be represented under TC - Representing the user community to the broader community (Board of directors, TC...) - Working with user groups worldwide to keep the community vibrant and informed. 3- The current active workgroups under UC’s governance will be moved to be under TC’s governance. 4- AUC and ATC to be merged into a single list, potentially called AC (Active Contributor) 5- We hope to have the merger finalized before the upcoming UC elections in September 2020. In addition to discussions over the mailing list, we also have the opportunity of "face-to-face" discussions at the upcoming PTG. Thanks Mohamed --melsakhawy

8 11

[blazar][ptg] IRC meetings cancelled due to PTG
by Pierre Riteau 28 May '20

28 May '20

Hello, The Blazar community will meet several times through the PTG event next week. Let's cancel the IRC meetings originally scheduled on June 2 and June 4. Talk to you soon, Pierre Riteau (priteau)

1 0

[glance] "true is not of type boolean" when creating protected image
by Bernd Bausch 28 May '20

28 May '20

This problem occurs on a stable Ussuri Devstack. I issue this command to create a protected image: $ openstack image create myimage2 --file ~/myvm1.qcow2 --disk-format qcow2 --protected and get this error message back: BadRequestException: 400: Client Error for url: http://192.168.1.200/image/v2/images, 400 Bad Request: On instance['protected']:: 'type': 'boolean'}: {'description': 'If true, image will not be deletable.',: 'True': Provided object does not match schema 'image': 'True' is not of type 'boolean': Failed validating 'type' in schema['properties']['protected']: It seems that image creation fails because 'True' is not of type 'boolean' Am I looking at a bug or am I doing something wrong? I do have a great workaround: First, create the image without the protected flag. Then: $ openstack image set --protected myimage2 This works.

2 1

[all] [kolla] Kolla Builder - next generation
by Radosław Piliszek 28 May '20

28 May '20

Hello fellow OpenStackers and Kollars (or Koalas) in particular, today is the day when I finally sit down to write up my thoughts on Kolla, and I mean *Kolla*, the container image building project for OpenStack and friends, nothing more (nor less). *** Some background on Kolla to get everyone on the same page (or close enough): Kolla builds images for production use. Kolla is upstream for TripleO containers and Kolla-Ansible deployment. These are container images, think OCI and Docker in particular since Kolla actually relies on Dockerfile format to specify build recipes and Docker runs building (TripleO runs buildah on the same recipes). Kolla supports three distributions as bases: CentOS (TripleO does override this for RHEL as well), Debian and Ubuntu. In the interim periods Kolla supports two distribution releases to ease/smooth the transition process for operators (like currently CentOS 7 and 8 in Train, while Ussuri is 8 only). What is more, each distro has two flavours (or 'types' as they are called for now): binary and source. This 'type' applies only to OpenStack software. Binary means that Kolla uses downstream packages (rpm/deb) while source means to use pip and install from official tarballs (or repos if it is master) and PyPI for deps, utilising venv to keep it separate from distro stuff. Finally, there is particular target architecture: x86_64, aarch64 and ppc64le. All of above affect the support matrix which is based on 3 or 4 dimensions (distro, flavour, arch and sometimes distro version). [1] Kolla offers a high level of customisability via various overrides levels. See docs for details [2] Kolla engine offers hierarchical approach to image building, under the assumption that more than one image is often deployed on the same machine so layer sharing is beneficial. Kolla helps collect sources for OpenStack projects to build desired versions of them. All recipes are templated using Jinja2 syntax. Images contain both run-time and, mostly in the case of 'source' flavour, build-time tools and libraries. *** That's it for the background. :-) So where is the problem you might ask? Oh, there are plenty. The general is that Kolla has lots of logic in templates. Thankfully, Kolla has macros for most stuff but still Jinja2 limitations make it hard to document exceptions (no inline comments in arrays anyone?). We lose visibility into real dependency graphs and may be easily reinstalling same stuff, any optimisations are limited and would require parsing of both Jinja2 and Dockerfile syntax. In general the current approach is ugly and becomes unwieldy (e.g. getting warnings about empty continuation lines). There is layering hell: dependency on long &&ed commands which are templated out to avoid useless layers. Support matrix is great for an overview. There is a hidden layer to it though, different combinations may support different features. This is not documented so far and not so easy to follow from sources. This stems from the fact that different extra components have different availability in distros, but this is not easily apparent from sources. Also, there is no way to turn feature on/off. You might do an override, but then again, which to keep, which not to? Which is which? Oh! Finally, the images tend to be heavy due to inclusion of build-time deps. *** That's it for issues (hopefully!) My idea is to reuse Kolla engine where it shines: sources collection, system of plugins, hierarchical building; but replace the part that smells - Jinja2 templating. Giving up on Jinja2 might encompass giving up on Dockerfile syntax, but that is optional and depends on what makes it more amenable to avoid further pitfalls. With giving up on Jinja2, the idea is to generate building recipes fully programmatically from Python. It would be possible to introduce "Features" - sets of packages to install based on the (distro, arch) tuple. This would result in more flexibility - turning them on/off (some could be optional, some not). There could be more than one optimization strategy regarding when packages get installed: you want only standalone blah-blah? Then Kolla won't be installing XYZ and ABC just because ugma-ugma and tickle-tickle require them and you "could save some space" (TM). In the same vein, features could declare which components are build-time and which are run-time and this would make it straightforward to separate the sides. The above effort could well be coordinated with different projects to reuse bindep contents. So far Kolla does not use bindep because it often installs too much and not enough at the same time. Do note it would still be bindep-less for external services. There would still need to exist a general mechanism for providing custom command executions required by some images. For contributors and cores this new approach would bring more sanity as to the scope of proposed changes. Also, it would be possible to get quick insight into feature support and autogenerate docs for that as well. Similarly, current concept of unbuildable images would no longer be required because unbuildability would be dictated by lack of support for a required feature. *** Looking forward to your opinions/thoughts. *** [1] https://docs.openstack.org/kolla/train/support_matrix.html [2] https://docs.openstack.org/kolla/train/admin/image-building.html -yoctozepto

5 9

Re:[Cyborg]Edge sync up at the PTG
by yumeng bao 28 May '20

28 May '20

Hi Ildikó, Thanks for the post! Cyborg team would be glad to join ! I have added this edge group meeting session to our PTG schedule : https://etherpad.opendev.org/p/cyborg-victoria-goals Please help to add links for the discussion if there is any. Regards， Yumeng

2 2

[kolla] Virtual PTG
by Mark Goddard 28 May '20

28 May '20

Hi, Just a reminder that it's the virtual PTG next week. We have an Etherpad [1] which has lots of information, and remember to register (free) [2] if you haven't already. Kendall sent some useful information [3] recently also. If you plan to attend, please add your name to the Etherpad so we know who to expect for each session (although you are welcome to turn up if you have not). The Etherpad is currently quite light on proposed discussion topics. Please do spend some time to consider what we should be using this time to discuss. It doesn't have to be a shiny new feature, it could be a long standing bug that we should agree on how to address, or a discussion on how to improve our processes. If we are left with time to spare however, I think it will be well spent on some reflection, maintenance, and planning. Cheers, Mark [1] https://etherpad.opendev.org/p/kolla-victoria-ptg [2] https://virtualptgjune2020.eventbrite.com/ [3] http://lists.openstack.org/pipermail/openstack-discuss/2020-May/015054.html

1 0

[sdk][osc][cli][ptg][project_cleanup] PTG Week planning
by Artem Goncharov 28 May '20

28 May '20

Hi everyone interested :) The PTG planning for the OpenStackSDK and the CLI is there: https://etherpad.opendev.org/p/SDK-VictoriaPTG-Planning <https://etherpad.opendev.org/p/SDK-VictoriaPTG-Planning> Please enter yourself there if interested and add topics you want to discuss. We have currently 2 slots: 01 June: 15:00 - 17:00 UTC Juno 03 June: 15:00 - 17:00 UTC Bexar And we plan to be using meetpad (fingers crossed). Interesting topics: - June 01 15:00-16:00 project cleanup - long standing topic across the projects - June 01 16:00-17:00 openstackclient-first - a one time failed community goal, and now maybe multi-release goal for joining all individual service clients into OSC to give users seamless UX Hear from you soon, Artem

2 1

[glance][ptg] Victoria Virtual PTG updates
by Abhishek Kekane 28 May '20

28 May '20

Hello All, Greetings!!! First virtual PTG is around the corner and if you haven't already registered, please do so as soon as possible [1]. I have created a Virtual PTG planning etherpad [2] and also added day wise topics along with timings we are going to discuss. Kindly let me know if you have any concerns with allotted time slots. As a reminder, these are the time slots for our discussion. Monday 01 June 2020 1400 UTC to 1700 UTC Folsom Tuesday 02 June 2020 2100 UTC to 00 UTC Austin Wednesday 03 June 2020 2100 UTC to 00 UTC Austin Thursday 04 June 2020 2100 UTC to 00 UTC Folsom Friday 05 June 2020 1400 UTC to 1700 UTC Folsom We will be using meetpad [3] for our discussion, kindly try to use it once before the actual discussion. I will publish the meeting URL before the PTG and which will be the same throughout the PTG. [1] https://virtualptgjune2020.eventbrite.com/ [2] https://etherpad.opendev.org/p/Glance-Victoria-PTG-planning [3] https://meetpad.opendev.org Thank you, Abhishek

1 0

[manila][ptg] Victoria Virtual PTG Items
by Goutham Pacha Ravi 28 May '20

28 May '20

Hello Zorillas and other interested Stackers, If you haven't already registered to be part of the virtual PTG, please do so as soon as possible [1]. There's now an etherpad posted to gather topics for the Ussuri Team Retrospective [2]. Please take a look and add your thoughts before the event. Also take a look at our Virtual PTG planning etherpad [3], and let me know if you have a problem with the draft schedule posted. As a reminder, these are the time slots for our meeting: == Monday, 1st June 2020 == 1400 UTC - 1700 UTC == Tuesday, 2nd June 2020 == 1500 UTC - 1700 UTC 2100 UTC - 2300 UTC == Friday, 5th June 2020 == 1400 UTC - 1700 UTC Our meetings are on Zoom and will be recorded. The Zoom Room is called "Cactus" and is linked out of the PTG Schedule page [4]. So make sure to bookmark it :) Looking forward to a happy week of designing with you all! -- Goutham Ravi [1] https://virtualptgjune2020.eventbrite.com/ [2] https://etherpad.opendev.org/p/manila-ussuri-retrospective [3] https://etherpad.opendev.org/p/vancouver-ptg-manila-planning [4] http://ptg.openstack.org/ptg.html

1 0

[nova][ptg] Healthchecks API
by Ghanshyam Mann 28 May '20

28 May '20

Hello Everyone, This is one of the PTG topic[1] but it will be good if we can get some discussion to happen before PTG and utilize the PTG time. I am sorry for starting this thread late. To provide the healthchecks API for nova, I have done 1 PoC which is basically extending the oslo healthchecks middleware plugins to add the real checks. - https://review.opendev.org/#/c/731396/ I am writing the details on this etherpad[2] where you can see the response and the flow of new plugins. Also describing briefly here: oslo provides healthchecks middleware with plugin framework. oslo provides two basic plugins which does file/port based health checks - disable_by_file - disable_by_files_ports New Idea is to extend those checks with new plugins on nova side: Nova can provide three plugins which are configurable[3]: 1. Nova_DB_healthcheck: Checks if API, cell0 and at least one cell DB is up. If so then return Healthy otherwise Unhealthy 2. Nova_MQ_healthcheck: Checks if at least one cells MQ is up. If so then return Healthy otherwise Unhealthy 3. Nova_services_healthcheck: Checks if at least one cell has at least one conductor and one compute service running. If so then return Healthy otherwise Unhealthy All plugins will return the dict of results for example DB dict of API, all cells DB with status, Please refer the example response in later part. TODO: Auth part for various plugins Flow diagrame of new plugins: - Nova_DB_healthcheck: -->API DB is up | --> cell0 DB and at least one cell DB is up I --> Return OK - Nova_MQ_healthcheck: -->API MQ is up | --> At leask: one cell MQ is up I --> Return OK - Nova_services_healthcheck: --> At least one cell has at least one conductor and one compute running --> TODO: need to check other services for example scheduler at least I --> Return OK Result: 200 OK if all enabled plugins return OK otherwise 503 Service Unavailable. Opinion/thoughts? [1] https://etherpad.opendev.org/p/nova-victoria-ptg [2]https://etherpad.opendev.org/p/nova-healthchecks [3] https://review.opendev.org/#/c/731396/1/etc/nova/api-paste.ini@107

1 0

[sdk][api-sig][ptg] Victoria PTG planning
by Artem Goncharov 28 May '20

28 May '20

Hi everyone interested :) The PTG planning for the API-SIG is there: https://etherpad.opendev.org/p/API-SIG-VictoriaPTG-Planning <https://etherpad.opendev.org/p/SDK-VictoriaPTG-Planning> Please enter yourself there if interested and add topics you want to discuss. We have currently 1 slot, so hurry up not to loose the chance: 02 June: 16:00 - 17:00 UTC Liberty And we plan to be using meetpad (fingers crossed). Hear from you soon, Artem

2 1

[all][interop WG] Call for Participationin for Virtual PTG interop WG and seek ambassadors
by prakash RAMCHANDRAN 27 May '20

27 May '20

Hi all, We are looking for participation & presenters, contributors form different OpenStack and adjacent projects in Open Infrastructure , k8s, CNCF, OPNFV, ONAP, OCN etc. Please register in etherpad and review for anything you like to share wrt user level API for Multi Cloud, Hybrid Cloud and Edge cloud Interop capability or compatibility. The bridge will be tested this Friday May 29th before we finalize the location or bridge, Add you name topic and your inputs for rapid delivery slots of say 5-10 mts each on June 1st. https://etherpad.opendev.org/p/interop_virtual_ptg_planning_june_2020 Schedule June 1 (UTC 13-15 hrs = 6AM-8 AM PDT = 9AM-11 AM EST = 9PM-11PM BJT) First hr will focus on Project team like Nova, Neutron, Ciner, Keystone, Glance & Swift.Second Hour - Open for new ideas Additionally we would like ambassadors who can register and attend other Virtual PTG projects and provide us feedback after attending them as what they can provide to us for Interop WG from all other projects. ThanksFor Interop WGChair PrakashVice Chair Mark

1 0

[tripleo] Ussuri containers
by Wesley Hayutin 27 May '20

27 May '20

Greetings, FYI we had a squatter in docker.io steal away the tripleoussuri namespace. We are investigating using quay but we need to do quite a bit of testing for it. For now please remind folks that the default container registry namespace is not tripleo$release for ussuri. We're using "tripleou" [1]. If you want to pull from the rdo registry, you will find those containers in tripleoussuri for now [2], but will be moved to use tripleou as well for consistency. Note the following update to tripleo-common as well [3] Full story is that we attempted to contact docker.io support several times as they have a policy around removing squatting namespaces. The support team never responded. Our goal is drop docker.io and use quay.io but taking a step by step phased approach to that. Thanks all [1] https://hub.docker.com/u/tripleou [2] https://registry.rdoproject.org:8443/oapi/v1/namespaces/tripleoussuri/image… [3] https://review.opendev.org/#/c/731244/

1 0

[Nova][CentOS] extrem slow first boot Centos 8.1 with failures
by Nicolas Ghirlanda 27 May '20

27 May '20

Hello all, I created a centos image with a pretty similar kickstart config like from that link https://github.com/CentOS/sig-cloud-instance-build/pull/159/commits/2c542bb… Packer is building the image fine, also the post installation tasks after a reboot in qemu (which is running fine and fast), so everything looks good so far. Then upload to openstack and on first boot, the instance is booting fine until that point of boot: [[0;32m OK [0m] Started D-Bus System Message Bus. [[0;32m OK [0m] Started Hardware RNG Entropy Gatherer Daemon. Starting OpenSSH rsa Server Key Generation... Starting System Security Services Daemon... [ 9.913211] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 9.988763] input: PC Speaker as /devices/platform/pcspkr/input/input6 [ 10.137969] sd 0:0:0:0: Attached scsi generic sg0 type 0 [ 10.332234] cirrus 0000:00:02.0: vgaarb: deactivate vga console [ 10.419654] Console: switching to colour dummy device 80x25 [ 10.421488] [TTM] Zone kernel: Available graphics memory: 2017692 kiB [ 10.422946] [TTM] Initializing pool allocator [ 10.424036] [TTM] Initializing DMA pool allocator [ 10.425630] [drm] fb mappable at 0xFC000000 [ 10.426612] [drm] vram aper at 0xFC000000 [ 10.427436] [drm] size 33554432 [ 10.428108] [drm] fb depth is 16 [ 10.428884] [drm] pitch is 2048 [ 10.449202] fbcon: cirrusdrmfb (fb0) is primary device [ 10.465230] Console: switching to colour frame buffer device 128x48 [ 10.499394] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 10.524033] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 On the console there is no further update from here. In the boot log it slowly progresses, but with Failures of starting services e.g. Starting RPC Bind... Starting Security Auditing Service... [ 9.500362] audit: type=1400 audit(1590589814.846:4): avc: denied { read } for pid=784 comm="auditd" name="group" dev="sda1" ino=259654 scontext=system_u:system_r:auditd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0 [ 9.517988] audit: type=1400 audit(1590589814.863:5): avc: denied { read } for pid=783 comm="rpcbind" name="passwd" dev="sda1" ino=259634 scontext=system_u:system_r:rpcbind_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0 [[0;1;31mFAILED[0m] Failed to start RPC Bind. See 'systemctl status rpcbind.service' for details. or [[0;32m OK [0m] Started OpenSSH ecdsa Server Key Generation. [[0;32m OK [0m] Started GSSAPI Proxy Daemon. [[0;32m OK [0m] Started OpenSSH ed25519 Server Key Generation. [[0;32m OK [0m] Started OpenSSH rsa Server Key Generation. [[0;1;31mFAILED[0m] Failed to start Authorization Manager. See 'systemctl status polkit.service' for details. [[0;1;33mDEPEND[0m] Dependency failed for Dynamic System Tuning Daemon. [[0;1;31mFAILED[0m] Failed to start NTP client/server. See 'systemctl status chronyd.service' for details. Also cloud-init is not able to assign an ip. [[0;32m OK [0m] Started D-Bus System Message Bus. Starting Network Manager... [ 550.962041] cloud-init[1070]: Cloud-init v. 18.5 running 'init' at Wed, 27 May 2020 14:37:46 +0000. Up 460.76 seconds. [ 550.964516] cloud-init[1070]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++ [ 550.967275] cloud-init[1070]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+ [ 550.970075] cloud-init[1070]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | [ 550.972524] cloud-init[1070]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+ [ 550.975221] cloud-init[1070]: ci-info: | ens3 | False | . | . | . | fa:16:3e:a3:7d:d5 | [ 550.977970] cloud-init[1070]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . | [ 550.980623] cloud-init[1070]: ci-info: | lo | True | ::1/128 | . | host | . | [ 550.983300] cloud-init[1070]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+ [ 550.986164] cloud-init[1070]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++ [ 550.988469] cloud-init[1070]: ci-info: +-------+-------------+---------+-----------+-------+ [ 550.991092] cloud-init[1070]: ci-info: | Route | Destination | Gateway | Interface | Flags | [ 550.993576] cloud-init[1070]: ci-info: +-------+-------------+---------+-----------+-------+ [ 550.996326] cloud-init[1070]: ci-info: +-------+-------------+---------+-----------+-------+ After that time, the system is up, but no hostname is set (due to cloud-init issues). If I now reboot, everything is working fine, cloud-init is working etc. If I set selinux from enforcing to permissive in the kickstart config, the issues are not showing up, but I need selinux to stay enforcing. Anyone with the same issues or any hints appreciated. regards Nicolas

1 1

[ironic] [ptg] PTG week schedule
by Dmitry Tantsur 27 May '20

27 May '20

Hi ironicers! The PTG is next week, and I'd like to share our plans for it. We don't have anything planned on Monday, and the weekly meeting is cancelled as well. 2-hour slots are reserved Tuesday to Friday 14:00 UTC - 16:00 UTC in the Liberty room. The up-to-date schedule and notes are available on https://etherpad.opendev.org/p/Ironic-VictoriaPTG-Planning, the current version is as follows: Tue, Jun 2nd: 14:00-15:00: Releases, perceptions and our place in the world: governance and perceptions 15:00-16:00: Releases, perceptions and our place in the world: standalone future Wed, Jun 3rd: 14:00-15:00: Deployment Efficency 15:00-16:00: Aborting non-waiting transient states Thu, Jun 4th: 14:00-15:00: Fishy future 15:00-16:00: IPAM integrations outside of Neutron Fri, Jun 5th: 14:00-15:00: Operator suggestions/feedback/painpoints 15:00-16:00: Ideas and brainstorming I would like to highlight the operators feedback slot on Friday at 14:00 UTC. We would like to hear more opinions from people running ironic! Please join us. Hear from you soon, Dmitry

1 0

[kolla-ansible] External Ceph and cinder-volumes
by Alfredo De Luca 27 May '20

27 May '20

Hi all. I am sing kolla-ansible 4.0 with and external ceph already deployed. I have in globals.yml the following setting: enable_ceph: "no" glance_backend_ceph: "yes" nova_backend_ceph: "yes" cinder_backend_ceph: "yes" enable_cinder_backend_lvm: "no" When I run kolla-ansible I got an error about not having cinder-volumes group Cannot process volume group cinder-volumes Do you know why it keeps looking for the cinder volumes? I ve done it before with no problem but maybe something in the global not correctly configured? Cheers -- */Alfredo*

2 7

[cyborg] Edge sync up at the PTG
by Ildiko Vancsa 27 May '20

27 May '20

Hi Cyborg Team, I’m reaching out to check if you are available during next week to continue discussions on edge related topics that we started in Shanghai. The OSF Edge Computing Group is meeting next week and we have a slot reserved to sync up with OpenStack projects such as Cyborg on Wednesday (June 3) at 1300 UTC - 1400 UTC. Would the team be available to join at that time? Thanks, Ildikó

1 0

[tripleo][operators] Removal of mistral from the TripleO Undercloud
by Kevin Carter 27 May '20

27 May '20

Hello stackers, In the pursuit to remove Mistral from the TripleO undercloud, we've discovered an old capability that we need to figure out how best to handle. Currently, we provide the ability for an end-user (operator / deployer) to pass in "N" Mistral workflows as part of a given deployment plan which is processed by python-tripleoclient at runtime [0][1]. From what we have documented, and what we can find within the code-base, we're not using this feature by default. That said, we do not remove something if it is valuable in the field without an adequate replacement. The ability to run arbitrary Mistral workflows at deployment time was first created in 2017 [2] and while present all this time, its documented [3] and intra-code-base uses are still limited to samples [4]. As it stands now, we're on track to making Mistral inert this cycle and if our progress holds over the next couple of weeks the capability to run arbitrary Mistral workflows will be the only thing left within our codebase that relies on Mistral running on the Undercloud. So the question is what do we do with functionality. Do we remove this ability out right, do we convert the example workflow [5] into a stand-alone Ansible playbook and change the workflow runner to an arbitrary playbook runner, or do we simply leave everything as-is and deprecate it to be removed within the next two releases? Although it would be perfectly acceptable to keep Mistral around a little while longer, it would also be a bummer if this one capability was the only reason we were not able to remove Mistral from the Undercloud. Any and all feedback from operators, deployers, developers, former developers, etc would be greatly appreciated. Thank you in advance. [0] https://opendev.org/openstack/python-tripleoclient/src/commit/af719b795bc9b… [1] https://opendev.org/openstack/python-tripleoclient/src/commit/af719b795bc9b… [2] https://review.opendev.org/#/c/457874 [3] https://specs.openstack.org/openstack/tripleo-specs/specs/pike/tripleo-deri… [4] https://opendev.org/openstack/tripleo-heat-templates/src/commit/48da6a13936… [5] https://opendev.org/openstack/tripleo-common/src/commit/cc4f9c57b708259e9e9… Kevin Carter IRC: kecarter

5 9

[tripleo] TripleO Ussuri has been released
by Wesley Hayutin 27 May '20

27 May '20

Greetings, I'm pleased to announce that TripleO Ussuri has been released today. Special thanks to Chandan Kumar and Emilien Macchi for the extra time and effort in getting everything in place for the release. Thanks to everyone else as well for your contributions, bug reports, patches and participation! Some highlights from the TripleO release: * The mistral to ansible conversion blueprint is complete * The nova-less-deploy feature is complete * The TripleO-Operator-Ansible feature is complete * The RDO and CI teams completed work on component based DLRN rpm repos and CI. * TLS and TripleO deployments tested together in upstream CI. * Support for CentOS-8 * Many other improvements and updates Congratulations to the following individual contributors for their reviews across ALL openstack projects!! Alex Schultz, the #1 reviewer for all projects in Ussuri Herve Beraud, the #3 reviewer for all projects in Ussuri Emilien Macchi, the #4 reviewer for all projects in Ussuri Followed closely behind in reviews by Stephen Finucane Dmitry Tantsur Slawek Kaplonski Thanks to all our TripleO contributors!! https://www.stackalytics.com/?release=ussuri&metric=commits&project_type=op… Oh by the way TripleO fixed 558 bugs this release as well :) Additional statistics can be found here: https://docs.google.com/presentation/d/1RV30OVxmXv1y_z33LuXMVB56TA54Urp7oHI… <https://docs.google.com/presentation/d/1RV30OVxmXv1y_z33LuXMVB56TA54Urp7oHI…> Release notes can be found per TripleO project: https://docs.openstack.org/releasenotes/ At the time of writing this email not all the release notes have been published. Thanks all and I'm looking forward to seeing you at the PTG!

2 1

[PTG][neutron] Draft of the agenda
by Slawek Kaplonski 27 May '20

27 May '20

Hi neutrinos, I just prepared draft of the agenda for the Neutron team for the upcoming virtual PTG. It's available in the etherpad [1]. Please take a look into it and reach out to me if You: * want to change day/time of topic(s) which You are interested in mostly due to e.g. some conflicts which You may have with some other meetings, * if You want to add some new topics which, to make sure that agenda is modified to include that, * if You see anything else which You thinks that should be changed there :) See You all on the virtual PTG next week :) [1] https://etherpad.opendev.org/p/neutron-victoria-ptg -- Slawek Kaplonski Senior software engineer Red Hat

1 0

[all][policy] Secure Default Policies Popup team meeting
by Ghanshyam Mann 27 May '20

27 May '20

Hello Everyone, To speed up the work on Secure Default Policies Popup team[1], we are going to start the biweekly meeting to discuss the pending things to finish if any or project need help on implementing the new default policy. We encourage projects interested to implement the new defaults or at least those listed on the wiki page [2] to join that meeting. Meeting details are on the wiki[3] also I have proposed the patch to book the slot: https://review.opendev.org/#/c/730935/ Agenda is slightly open for now which we will be moving to concrete as we progress and timing is biweekly-even on Thursday starting from this week on 28th 1800 UTC. We can adjust the timing also if needed. NOTE: First meeting on this week Thursday 28th 1800UTC on #openstack-meeting [1] https://governance.openstack.org/tc/reference/popup-teams.html#secure-defau… [2] https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… [3] https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… -gmann

1 0

[openstack-helm] Stepping down as core reviewer
by Steve Wilkerson 26 May '20

26 May '20

Hey everyone, I hope you're all staying safe and well. I switched jobs a few months back that took me in a different direction from my previous employer, and as a result I've not had the time to remain involved with the OpenStack community (especially as my day job no longer involves working with OpenStack). I think it's best that I step down from the openstack-helm core reviewer team, as I can't realistically make the time to do my due diligence in providing thorough, useful code reviews. I greatly appreciate the opportunities that working with the OpenStack community at large has provided me over the past 5 years. Cheers, srwilkers

2 1

[kolla-ansible][kolla][cinder]Cinder volume is not attached to instance
by Sanjay K 26 May '20

26 May '20

Hello Openstackers, I have used Cinder stein release (v14.0.2) source base and deployed in All-In-One setup on ubuntu16.04 using Kolla-Ansible. The cinder image was built using python 2.7 somewhere which i am trying to deploy in my new setup having python3.5. All services are deployed and running fine. However when I try to attach one volume to an instance, it shows below error in /var/log/kolla/cinder/cinder-volume.log. 2020-05-26 08:10:07.026 35 INFO oslo.privsep.daemon [req-db287360-876c-425e-b8df-a4f3b0db3163 bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'cinder.privsep.sys_admin_pctxt', '--privsep_sock_path', '/tmp/tmpan8c8r/privsep.sock'] 2020-05-26 08:10:10.952 35 INFO oslo.privsep.daemon [req-db287360-876c-425e-b8df-a4f3b0db3163 bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Spawned new privsep daemon via rootwrap 2020-05-26 08:10:10.647 1071 INFO oslo.privsep.daemon [-] privsep daemon starting 2020-05-26 08:10:10.721 1071 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2020-05-26 08:10:10.725 1071 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_NET_ADMIN|CAP_SYS_ADMIN/none 2020-05-26 08:10:10.726 1071 INFO oslo.privsep.daemon [-] privsep daemon running as pid 1071 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server [req-db287360-876c-425e-b8df-a4f3b0db3163 bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Exception during message handling: ProcessExecutionError: Unexpected error while running command. Command: tgtadm --lld iscsi --op show --mode target Exit code: 107 Stdout: u'' Stderr: u'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 4517, in attachment_update 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server connector) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 4445, in _connection_create 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server volume, connector) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 830, in create_export 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server volume_path) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 210, in create_export 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server **portals_config) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/utils.py", line 818, in _wrapper 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 206, in call 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return attempt.get(self._wrap_exception) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 247, in get 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2]) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 200, in call 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/targets/tgt.py", line 126, in create_iscsi_target 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server (out, err) = cinder.privsep.targets.tgt.tgtadm_show() 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 244, in _wrap 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 203, in remote_call 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server raise exc_type(*result[2]) 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command. 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Command: tgtadm --lld iscsi --op show --mode target 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Exit code: 107 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Stdout: u'' 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server Stderr: u'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' 2020-05-26 08:10:11.550 35 ERROR oslo_messaging.rpc.server When I check inside cinder-volume container, the tgtd is not running. SO I started tgtd from root user. Now the volume attachment happens for few seconds and then it is detached. 2020-05-26 08:14:39.221 35 INFO cinder.volume.manager [req-5da81388-33ce-4994-9114-52442a08ac2a bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] attachment_update completed successfully. 2020-05-26 08:14:52.622 35 INFO cinder.volume.manager [req-cbaa1b6c-4fa8-4ece-b08b-8ca09f78ecaf bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Terminate volume connection completed successfully. 2020-05-26 08:14:52.840 35 INFO cinder.volume.targets.tgt [req-cbaa1b6c-4fa8-4ece-b08b-8ca09f78ecaf bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] Removing iscsi_target for Volume ID: d713ce55-1fa6-4d96-a69e-69368494f8f9 2020-05-26 08:14:53.739 35 WARNING py.warnings [req-cbaa1b6c-4fa8-4ece-b08b-8ca09f78ecaf bc13e1a279854a6fa1d553c7b7a0214c 9bdfc4ffefaa4edb823ff9ba5383ab42 - default default] /var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/evaluator.py:99: SAWarning: Evaluating non-mapped column expression 'updated_at' onto ORM instances; this is a deprecated use case. Please make use of the actual mapped columns in ORM-evaluated UPDATE / DELETE expressions. "UPDATE / DELETE expressions." % clause I also used the cinder 14.0.4 source tag to build the new image, but the same error persists. Can you please point me what is the issue in my test case or setup. Appreciate your help. best regards, Sanjay

2 1