May 2022 - openstack-discuss - lists.openstack.org

[neutron] Drivers meeting - Friday 6.5.2022 - cancelled
by Lajos Katona 05 May '22

05 May '22

Hi Neutron Drivers! Due to the lack of agenda, let's cancel tomorrow's drivers meeting. See You on the meeting next week. Lajos Katona (lajoskatona)

1 0

Magnum metrics-server not working
by Oliver Weinmann 05 May '22

05 May '22

Hi, Is the metrics-server broken by default in magnum? I tried for several hours to get it running, but running kubectl top nodes fails. Message: failing or missing response from https://10.100.2.4:8443/apis/metrics.k8s.io/v1beta1: Get "https://10.100.2.4:8443/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I also googled a lot but only found similar issues for k8s clusters not deployed via magnum. I remembered from my other cluster that I had to enable enable-aggregator-routing=true in kupe-apiserver and --kubelet-insecure-tls for the metrics-server. But this also doesn’t help. Any help would be highly appreciated. Best regards, Oliver

2 3

RE: Data Center Survival in case of Disaster / HW Failure in DC
by CHALANSONNET Stéphane (Acoss) 05 May '22

05 May '22

Hello, First Openstack isnt VMware.Crtiticals applications need to be design with a multiples regions/AZ aware. Best way was two regions , 2 contrôle plane (keystone , Galera cluster on both (not the same) 2 storage system and application spitted on both is the best pratice 😉 You can do also with two AZ but the control plane is share between two AZ (galezra cluster and RabbitMq ) If you are trying to do the same thing as a streched VMware Cluster on a dual-site it's complicated, i you have 3 sites (low latencies <1ms) you can ! 1.In case of any Sudden Hardware failure of one or more controller node OR Compute node OR Storage Node what will be the immediate redundant recovery setup need to be employed ? => You need to splt control plane throught multiples AZ (3 is the best for Galera cluster and RabbitMQ , 2 if you have HA VMware) => Network compute work with keepalived so also need to split them on two AZ/region, you need . Dhcp , Metadata services are also redundancy on multiples network node => compute : HA OpenStack (like VMware HA) seems to work since Wallaby with Masakari . Actually you need to do the job by yourself if you need to failover instances. The storage is also the bggest problem . You need a streched storage that work with Cinder : Ceph can do the job but you also need three site (quorum on monitor need 2 node alives).Netapp solution seems to work well (trident NFS) 2. In case H/W failure our recovery need to as soon as possible. For example less than30 Minutes after the first failure occurs. Galera Cluster and RabbitMq need to be check first With two sites 30mn is really a big challenge, with 3 site it should good ! 3. Is there setup options like a hot standby or similar setups or what we need to employ ? 2 AZ , 2 storage system and application spitted on both is the best pratice 😉 4. To meet all RTO (< 30 Minutes down time ) and RPO(from the exact point of crash all applications and data must be consistent) . Regards, Stéphane Chalansonnet -----Message d'origine----- De : openstack-discuss-request(a)lists.openstack.org <openstack-discuss-request(a)lists.openstack.org> Envoyé : jeudi 5 mai 2022 11:56 À : openstack-discuss(a)lists.openstack.org Objet : openstack-discuss Digest, Vol 43, Issue 12 Send openstack-discuss mailing list submissions to openstack-discuss(a)lists.openstack.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss or, via email, send a message with subject or body 'help' to openstack-discuss-request(a)lists.openstack.org You can reach the person managing the list at openstack-discuss-owner(a)lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Data Center Survival in case of Disaster / HW Failure in DC (KK CHN) 2. [nova][placement] Incomplete Consumers return negative value after upgrade (Jan Wasilewski) 3. Re: [all][tc][Release Management] Improvements in project governance (Slawek Kaplonski) ---------------------------------------------------------------------- Message: 1 Date: Thu, 5 May 2022 14:46:11 +0530 From: KK CHN <kkchn.in(a)gmail.com> To: openstack-discuss(a)lists.openstack.org Subject: Data Center Survival in case of Disaster / HW Failure in DC Message-ID: <CAKgGyB_UmDG+TknDaYLGAuyOnxY8ipSF3qC4tS90T0zHzu6fZg(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" List, We are having an old cloud setup with OpenStack Ussuri usng Debian OS, (Qemu KVM ). I know its very old and we can't upgrade to to new versions right now. The Deployment is as follows. A. 3 Controller in (cum compute nodes . VMs are running on controllers too..) in HA mode. B. 6 separate Compute nodes C. 3 separate Storage node with Ceph RBD Question is 1. In case of any Sudden Hardware failure of one or more controller node OR Compute node OR Storage Node what will be the immediate redundant recovery setup need to be employed ? 2. In case H/W failure our recovery need to as soon as possible. For example less than30 Minutes after the first failure occurs. 3. Is there setup options like a hot standby or similar setups or what we need to employ ? 4. To meet all RTO (< 30 Minutes down time ) and RPO(from the exact point of crash all applications and data must be consistent) . 5. Please share your thoughts for reliable crash/fault resistance configuration options in DC. We have a remote DR setup right now in a remote location. Also I would like to know if there is a recommended way to make the remote DR site Automatically up and run ? OR How to automate the service from DR site to meet exact RTO and RPO Any thoughts most welcom. Regards, Krish -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220505…> ------------------------------ Message: 2 Date: Thu, 5 May 2022 11:31:16 +0200 From: Jan Wasilewski <finarffin(a)gmail.com> To: openstack-discuss(a)lists.openstack.org Subject: [nova][placement] Incomplete Consumers return negative value after upgrade Message-ID: <CAN4DDNgG=i8tGpPQW2PUXVUp72T9r4CWDdxmfRg-bX8T1eE5AA(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hi, after an upgrade from Stein to Train, I hit an issue with negative value during upgrade check for placement: # placement-status upgrade check+------------------------------------------------------------------+| Upgrade Check Results |+------------------------------------------------------------------+| Check: Missing Root Provider IDs || Result: Success || Details: None |+------------------------------------------------------------------+| Check: Incomplete Consumers || Result: Warning || Details: There are -20136 incomplete consumers table records for || existing allocations. Run the "placement-manage db || online_data_migrations" command. |+------------------------------------------------------------------+ Seems that negative value is a result that I get such values from tables consumer and allocations: mysql> select count(id), consumer_id from allocations group by consumer_id;...1035 rows in set (0.00 sec) mysql> select count(*) from consumers;+----------+| count(*) |+----------+| 21171 |+----------+1 row in set (0.04 sec) Unfortunately such warning cannot be solved by execution of suggested command( placement-manage db online_data_migrations) as it seems it adds records to consumers table - not to allocations, which looks like to be a problem here. I was following recommendations from this discussion: http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018536… but unfortunately it doesn't solve the issue(even not changing a negative value). I'm just wondering if I skipped something important and you can suggest some (obvious?) solution. Thank you in advance for your time and help. Best regards, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220505…> ------------------------------ Message: 3 Date: Thu, 05 May 2022 11:55:32 +0200 From: Slawek Kaplonski <skaplons(a)redhat.com> To: openstack-discuss(a)lists.openstack.org Cc: El?d Ill?s <elod.illes(a)est.tech> Subject: Re: [all][tc][Release Management] Improvements in project governance Message-ID: <3198538.44csPzL39Z@p1> Content-Type: text/plain; charset="utf-8" Hi, On ?roda, 20 kwietnia 2022 13:19:10 CEST El?d Ill?s wrote: > Hi, > > At the very same time at the PTG we discussed this on the Release > Management session [1] as well. To release deliverables without > significant content is not ideal and this came up in previous > discussions as well. On the other hand unfortunately this is the most > feasible solution from release management team perspective especially > because the team is quite small (new members are welcome! feel free to > join the release management team! :)). > > To change to independent release model is an option for some cases, > but not for every project. (It is less clear for consumers what > version is/should be used for which series; Fixing problems that comes > up in specific stable branches, is not possible; testing the > deliverable against a specific stable branch constraints is not > possiblel; etc.) > > See some other comments inline. > > [1] https://etherpad.opendev.org/p/april2022-ptg-rel-mgt#L44 > > El?d > > On 2022. 04. 19. 18:01, Michael Johnson wrote: > > Comments inline. > > > > Michael > > > > On Tue, Apr 19, 2022 at 6:34 AM Slawek Kaplonski<skaplons(a)redhat.com> wrote: > >> Hi, > >> > >> > >> During the Zed PTG sessions in the TC room we were discussing some ideas how we can improve project governance. > >> > >> One of the topics was related to the projects which don't really have any changes in the cycle. Currently we are forcing to do new release of basically the same code when it comes to the end of the cycle. > >> > >> Can/Should we maybe change that and e.g. instead of forcing new release use last released version of the of the repo for new release too? > > In the past this has created confusion in the community about if a > > project has been dropped/removed from OpenStack. That said, I think > > this is the point of the "independent" release classification. > Yes, exactly as Michael says. > >> If yes, should we then automatically propose change of the release model to the "independent" maybe? > > Personally, I would prefer to send an email to the discuss list > > proposing the switch to independent. Patches can sometimes get > > merged before everyone gets to give input. Especially since the > > patch would be proposed in the "releases" project and may not be on > > the team's dashboards. > The release process catches libraries only (that had no merged > change), so the number is not that huge, sending a mail seems to be a fair option. > > (The process says: "Evaluate any libraries that did not have any > change merged over the cycle to see if it is time to transition them > to the independent release model > <https://releases.openstack.org/reference/release_models.html#openstack-rela…>. > Note: client libraries (and other libraries strongly tied to another > deliverable) should generally follow their parent deliverable release > model, even if they did not have a lot of activity themselves).") > >> What would be the best way how Release Management team can maybe notify TC about such less active projects which don't needs any new release in the cycle? That could be one of the potential conditions to check project's health by the TC team. > > It seems like this would be a straight forward script to write given > > we already have tools to capture the list of changes included in a > > given release. > > There are a couple of good signals already for TC to catch inactive > projects, like the generated patches that are not merged, for example: > > https://review.opendev.org/q/topic:reno-yoga+is:open > https://review.opendev.org/q/topic:create-yoga+is:open > https://review.opendev.org/q/topic:add-xena-python-jobtemplates+is:ope > n > > (Note that in the past not merged patches caused issues and discussing > with the TC resulted a suggestion to force-merge them to avoid future > issues) > > >> Another question is related to the projects which aren't really active and are broken during the final release time. We had such problem in the last cycle, see [1] for details. Should we still force pushing fixes for them to be able to release or maybe should we consider deprecation of such projects and not to release it at all? > > In the past we have simply not released projects that are broken and > > don't have people actively working on fixing them. It has been a > > signal to the community that if they value the project they need to > > contribute to it. > > Yes, that's a fair point, too, maybe those broken deliverables should > not be released at all. I'm not sure, but that might cause another > issues for release management tooling, though... > > Besides, during our PTG session we came to the conclusion that we need > another step in our process: > * "propose DNM changes on every repository by RequirementsFreeze (5 > weeks before final release) to check that tests are still passing with > the current set of dependencies" > Hopefully this will catch broken things well in advance. > > >> [1]http://lists.openstack.org/pipermail/openstack-discuss/2022-Marc > >> h/027864.html > >> > >> > >> -- > >> > >> Slawek Kaplonski > >> > >> Principal Software Engineer > >> > >> Red Hat > Thx for all inputs in that topic so far. Here is my summary and conclusion of what was said in that thread: * we shouldn't try automatically switch such "inactive" projects to the independent model, and we should continue bumping versions of such projects every cycle as it makes many things easier, * Release Management team will test projects about 5 weeks before final release - that may help us find broken projects which then can be discussed and eventually marked as deprecated to not release broken code finally, * To check potentially inactive projects TC can: * finish script https://review.opendev.org/c/openstack/governance/+/810037[1] and use stats generate by that script to periodically check projects' health, * check projects with no merged generated patches, like: https://review.opendev.org/q/topic:reno-yoga+is:open[2] https://review.opendev.org/q/topic:create-yoga+is:open[3] https://review.opendev.org/q/topic:add-xena-python-jobtemplates+is:open[4] Feel free to add/change anything in that summary if I missed or misunderstood anything there or if You have any idea about other improvements we can do in that area. -- Slawek Kaplonski Principal Software Engineer Red Hat -------- [1] https://review.opendev.org/c/openstack/governance/+/810037 [2] https://review.opendev.org/q/topic:reno-yoga+is:open [3] https://review.opendev.org/q/topic:create-yoga+is:open [4] https://review.opendev.org/q/topic:add-xena-python-jobtemplates+is:open -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220505…> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220505…> ------------------------------ Subject: Digest Footer _______________________________________________ openstack-discuss mailing list openstack-discuss(a)lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss ------------------------------ End of openstack-discuss Digest, Vol 43, Issue 12 *************************************************

1 0

[all][tc][Release Management] Improvements in project governance
by Slawek Kaplonski 05 May '22

05 May '22

Hi, During the Zed PTG sessions in the TC room we were discussing some ideas how we can improve project governance. One of the topics was related to the projects which don't really have any changes in the cycle. Currently we are forcing to do new release of basically the same code when it comes to the end of the cycle. Can/Should we maybe change that and e.g. instead of forcing new release use last released version of the of the repo for new release too? If yes, should we then automatically propose change of the release model to the "independent" maybe? What would be the best way how Release Management team can maybe notify TC about such less active projects which don't needs any new release in the cycle? That could be one of the potential conditions to check project's health by the TC team. Another question is related to the projects which aren't really active and are broken during the final release time. We had such problem in the last cycle, see [1] for details. Should we still force pushing fixes for them to be able to release or maybe should we consider deprecation of such projects and not to release it at all? [1] http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027864.ht… -- Slawek Kaplonski Principal Software Engineer Red Hat -------- [1] http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027864.ht…

4 4

[Kolla-ansible][xena] how to deploy multiple cinder backends?
by wodel youchi 05 May '22

05 May '22

Hi, I want to deploy multiple cinder backends, ceph SSD pool for high performance VMS, ceph SAS pool for normal VMS and NFS share for small VMS and for cinder backup. Could that be done while deploying or it has to be done after the initial deployment? Could you assist me with some examples, sometimes the documentation is not that obvious to understand. Regards.

2 1

[sdk] creating a blank volume with image properties
by John Ratliff 05 May '22

05 May '22

We're standing up a second openstack cluster to do a leapfrog migration upgrade. We are using the same ceph storage backend. I need to create volumes in the new openstack cluster that use the underlying ceph rbd. Currently, we are creating a volume in the new openstack cluster, deleting the rbd created by openstack, and renaming the original rbd to match the new openstack mapping. This process loses the image properties from the original volumes. We are using virtio-scsi for the disk block device, and want to continue doing that in the new cluster. When we create the server using the new volume in the new openstack cluster, it reverts to using virtio-blk (/dev/vda). What can I do to keep using our image properties in the newly created volume? I thought I could just pass volume_image_metadata as an argument to create_volume(), but it doesn't work. I end up with an image that has no image metadata properties. import openstack openstack.enable_logging(debug=False) conn = openstack.connect(cloud="test") v = conn.create_volume( name="testvol", size=5, wait=True, bootable=True, volume_image_metadata={ "hw_scsi_model": "virtio-scsi", "hw_disk_bus": "scsi", "hw_qemu_guest_agent": "yes", "os_require_quiesce": "yes", "hw_rng_model": "virtio", "image_id": "00000000-0000-0000-0000-000000000000", }, ) What can I do to create this openstack volume while preserving my original volume properties? Since we don't care about the image, I thought about creating a blank image that the volume could be based on with those properties, but I'm looking for suggestions. Thanks. --John

2 1

Re: [all][tc] Technical Committee next weekly meeting on May 5, 2022 at 1500 UTC
by Ghanshyam Mann 05 May '22

05 May '22

Hello Everyone, Below is the agenda for Tomorrow's TC IRC meeting schedule at 1500 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting == Agenda for tomorrow's TC meeting == * Roll call * Follow up on past action items * CFN overview from Jie Niu, China Mobile Research Institute (30 min) ** http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028345.ht… * 'tick', 'tock' release cadence (~30 min) ** Legal checks on using 'tick', 'tock' *** https://review.opendev.org/c/openstack/governance/+/840354 ** release notes discussion <rest of the agenda if time permits otherwise we will cover that in the next weekly meeting(12th May onwards)> * Join leadership meeting with Board of Directors * Gate health check ** Fixing Zuul config error in OpenStack *** https://etherpad.opendev.org/p/zuul-config-error-openstack * New ELK service dashboard: e-r service ** https://opensearch.logs.openstack.org/_dashboards/app/discover?security_ten… ** https://review.opendev.org/c/openstack/governance-sigs/+/835838 * Open Reviews ** https://review.opendev.org/q/projects:openstack/governance+is:open -gmann

1 0

Retiring and shutting down the ethercalc.openstack.org service
by Clark Boylan 04 May '22

04 May '22

Hello, The https://ethercalc.openstack.org service has become problematic for several reasons: * Our configuration management for the service is still based on Puppet and needs to be modernized to continue running the service. * The service itself isn't well maintained. Its build system is apparently broken [0]. * The service does not capture a history of changes like etherpad, confusing users. * The service crashes semi frequently. The major known user of this service was the PTG planning process. We've talked to the folks that help organize and plan the PTG, and they think other tools (like PTGBot) can be used instead without major trouble. Considering all of this, the OpenDev team would like to retire and shutdown this service. The current plan is to keep the service up and running until the end of the month. Then on May 31, 2022 shut down the server hosting the service, snapshot this server, and finally delete it. Please back up any data you would like to preserve before May 31, 2022. [0] https://github.com/audreyt/ethercalc/pull/781 If you have any questions or concerns feel free to respond to this thread or reach out to the OpenDev team. Clark

3 2

[horizon] Yoga: Syntax of new Parameter SYSTEM_SCOPE_SERVICES in local_settings.py
by Crandale 04 May '22

04 May '22

Hi there, We have started to analyze Openstack Yoga a bit and there, one of the major new feature is the activation of scope based token for regular use in nova. While after some long lasting back and forth in configuring our role assignments and policies we could make it work on one of our test environments (Ubuntu) via Openstack SDK, however, we are still struggling with some system scoped API calls to nova from horizon. We have an admin user for the domain 'Default' who has set the role ‚admin' for 'system all': +-------+---------------+-------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +-------+---------------+-------+---------+--------+--------+-----------+ | admin | admin@Default | | | | all | False | +-------+---------------+-------+---------+--------+--------+-----------+ We have configured in local_settings.py: SYSTEM_SCOPE_SERVICES = ['compute', 'image', 'volume', 'network‘] (Note: this config line has been reverse engineered from horizon source code as the syntax is nowhere possible to be found in the docs yet … - so: not sure if it is correct) Policy files are identical for horizon as for the services. For the user admin, we then get an additional field in the domain/project top line menu adding a ‚system scope‘ switch (this is what we understand how it should look like) and - when switching to system scope - also a system menu in the sidebar (also as expected). If we then go to System->Systeminformation to see the nova service list, we get an error ‚Unable to get nova services list‘, given reason is an error: 'Policy doesn’t allow os_compute_api:os-services:list to be performed (HTTP 403)‘. Informations for network and volume services are shown normally (here scoped tokens are not activated yet). Further analysis indicated that horizon is using still a project-scoped token and not a system-scoped one for these requests although ’system scope’ is active. Putting the same request from Openstack SDK with the same user admin results in $ openstack compute service list /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead from cryptography.utils import int_from_bytes /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead from cryptography.utils import int_from_bytes +----+------------------+-------------+----------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +----+------------------+-------------+----------+---------+-------+----------------------------+ | 4 | nova-consoleauth | controller | nova | enabled | down | 2019-10-31T14:59:33.000000 | | 5 | nova-scheduler | controller | nova | enabled | up | 2022-05-04T08:52:48.000000 | | 6 | nova-conductor | controller | nova | enabled | up | 2022-05-04T08:52:42.000000 | | 12 | nova-compute | compute3 | Crandale | enabled | up | 2022-05-04T08:52:40.000000 | | 13 | nova-conductor | controller3 | nova | enabled | down | 2020-06-28T14:45:31.000000 | | 14 | nova-scheduler | controller3 | nova | enabled | down | 2020-06-28T14:45:24.000000 | +----+------------------+-------------+----------+---------+-------+—————————————--------------—+ Which indicates that role assignments to user admin are correct. The same command with -—debug also proves that a system scoped token is generated. Before I consider to open a bug towards Horizon: Could someone indicate to me whether the syntax of the config needs some adaptions to make it work or confirm that it is correct? Is there any other aspect we overlooked? I am looking forward to your reply Best regards Clemens

2 1

[api=sig][i18n][infra][tc] Cleaning up more defunct mailing lists
by Jeremy Stanley 04 May '22

04 May '22

A quick audit of the mailing lists we host at lists.openstack.org turned up the following 11 which have had no new posts for at least three years now: * openstack-api-consumers * openstack-de * openstack-el * openstack-i18n-fr * openstack-ir * openstack-personas * openstack-ru * openstack-sos * openstack-tw * openstack-vi * third-party-announce I'd like to clean them up by removing their list configurations, but leaving any public archives intact. Are there any objections? Should any of the above have future messages redirected to a different list (though since they haven't had any for years, that's probably unnecessary)? -- Jeremy Stanley

5 8