June 2024 - openstack-discuss - lists.openstack.org

答复: [largescale-sig]scaling story
by Alex Song (宋文平) 27 Jun '24

27 Jun '24

0 *H÷ 010 +0 *H÷ $Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Hi Mohammed, The max number of conns for RabbitMQ is estimated through cacluating the conns of nova and cinder components. We have 3000 compute nodes, 3 control nodes and 15 cinder-volume nodes, We deploy rabbitmq on control nodes and use master-slave mode, the totally conns of RabbitMQ is almost 2w, so we set 2w for RabbitMQ configuration. Before deploying the SB relay, each ovn SB process needed to manage an average of 600+connections, the CPU usage is often 100% and the requests is slow processing. After adding the relay, each relay process handles about 60 connections, the total relay process is set to 10. After testing, each relay process has a low CPU usage and can process information quickly. -----é®ä»¶åä»¶----- åä»¶äºº: Thierry Carrez [mailto:thierry@openinfra.dev] åéæ¶éŽ: 2024å¹Ž6æ19æ¥ 17:29 æ¶ä»¶äºº: Alex Song (å®æå¹³) <songwenping(a)inspur.com> äž»é¢: Re: [largescale-sig]scaling story Hi Alex, Thanks for the great write up! I would love to see more of this. How did you adjust the max number of conns for RabbitMQ and for the relay I assume you used https://docs.ovn.org/en/latest/tutorials/ovn-ovsdb-relay.html <https://docs.ovn.org/en/latest/tutorials/ovn-ovsdb-relay.html> ? Thanks Mohammed *From: *Alex Song (å®æå¹³) <songwenping(a)inspur.com> *Date: *Wednesday, June 12, 2024 at 4:43â¯AM *To: *openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> *Subject: *[largescale-sig]scaling story Hi everyone, we want to share our scaling story. Please refer the link: https://etherpad.opendev.org/p/large-scale-inspur <https://etherpad.opendev.org/p/large-scale-inspur> Thanks, Alex Song 0É0± xðáwIÊèë`{0 *H÷ 0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA0 170109092830Z 340511122004Z0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA0"0 *H÷ 0 «ä5ïc$Œ©æ'µ¯Þ6>úUKÛdÔ²Áe9Î~{BîÒLgD÷*wvVÊŠ/DýUj_xá\m/ óž=kÐ·éGÙœQ€ýx~Wùgk ÛÜøãÔ7É6NçÏ*?n°Ê²mhùè{Ã¯ôÌÆ 7üF-Î<@ÃÓÍ¬WçÅyåLZrF 6~føÈ×T~$0d¡ýL|zšøW=ötÚ%ýq,¥Ã~Ÿ"ÀýŸÑö2T,QÕÔ,dºÂÅ^§ÈôïåJ)ëVvp Ó£00 +7CA0U0Uÿ0ÿ0U^YŠŽLX`Nöµ¥9Š2Á5j0 +70# +7&a°$öz(¶o§K0 *H÷ daòYÙ~×ì×NÑ3ŽËlP±i!¿ýÅòsÝLºP;I4hžØþ¡L¶äŠJêà·Õ:1àEÕs®9UÛ:8Mh{Ä²ìºŒ£·ÚŒó×÷C_ŸhoÏúÆËYM&UØ³?9êèr·ìgP_÷mÇ#YÖpîw0º¹«0$p³jºnº¥¯/Ôä¬÷Ú ŠÖ5uþQÝ¶h(y¶ènnã%E,øú&Zâ"dKØ³ãi5VúP9±œ/ñ õkØŸsá/Xðï»¯€µ³O÷Í-ó¿Mè³3Óž{G(ªŸßŒG0C0+ ~+mçbí++0 *H÷ 0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA0 191128064731Z 241126064731Z010 &ò,dcom10 &ò,dlangchao10 &ò,dhome10Uæµªæœ®ä¿¡æ¯10U å®æå¹³1%0# *H÷ songwenping@inspur.com0"0 *H÷ 0 ã4ÒEd$i÷18#,åH~èyµþJRáLêg¡$@6vÔ<sâelPO4ýooXZç1¢iéÁ IòKjÚO§h;Rfùê7Ÿp¬ŒæäÝPòüÿ~æ E}é{có» (ù'+zÞ s ,!K£;û¥¶DEu1+ðÏ»õØÛähJjŸj']a¥Î·7¡õòhõßHŒ8QzÆ{ËÍ73>, ªí|gœ0y€Xv'ÚÊ ¬ÀO#)³°øÕM^œ¢VñóŸ¯ìFñ;µ5HŽù£Ã0¿0= +700.&+7ò©×z©=÷Ø\Jý&§Md\0)U%"0 ++ +7 0U 05 +7 (0&0 +0 +0 +7 0D *H÷ 7050*H÷ 0*H÷ 0+0 *H÷ 0UW 0²^qûìÇUã¡b[]s0U#0^YŠŽLX`Nöµ¥9Š2Á5j0U00ÿ ü ùºldap:///CN=INSPUR-CA,CN=JTCA2012,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=home,DC=langchao,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint:http://JTCA2012.home.langchao.com/CertEnroll/INSPUR-CA.crl0)+00±+0€ldap:///CN=INSPUR-CA,CN=AIA,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=home,DC=langchao,DC=com?cACertificate?base?objectClass=certificationAuthority0a+0Uhttp://JTCA2012.home.langchao.com/CertEnroll/JTCA2012.home.langchao.com_INSPUR-CA.crt0IUB0@ & +7 songwenping@inspur.comsongwenping@inspur.com0 *H÷ §öãý+·>³â¿Xíe-÷¯[ÂÎ©É\ÃAæMà.U'ù||Wø$Šn' `9Ýµ¯éÉM^mz0Äð -AÒ¯×Tï+£§¡RbfzÆX#ùkY<ØDþxólõA%ÑøN¬×grÛÚ·ãF!GþóüVœÍ±Ð[ÏL®GÅ3€$ /DžÃiú_h|cI}DôþyªÉñ Ç}³W$_Äáy]ò±DÄ1(tr)6gêäÑæÄLd$!t{x±Z±·þÏeqmkþÒÒ÷r³ßxõ%lUÉÒ4d100p0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA~+mçbí++0 + ø0 *H÷ 1 *H÷ 0 *H÷ 1 240627081404Z0# *H÷ 1ŠÒ5KCuâ|!:¶!~ÏÚ/Ý0 +71r0p0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA~+mçbí++0*H÷ 1r p0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA~+mçbí++0 *H÷ 100 `He*0 `He0 *H÷ 0 `He0*H÷ 0 *H÷ @0+0 `He0 `He0 `He0 *H÷ Jû¡ûäž«$]ŽíijO yªLMÑ¢ÅúŸC±_U¶[1Jnl~/Aò¢\E gTŽ ENºÁ óª už6ÄµÕ/Ô/j/,ltµ~O¥Æù]Î0È9§DAªªC'8µG*9ÄíËüÛâèÔðHmá3ŽYËù*Ý'Ÿ¯ù§Ý,â9iâÙ2±×Á·ÞÌøtüoÙ wýÂ!7e&ß ëFÃ®T¢é3.k-îgøvc«nŠ*(G> óóFK°"óx aÙ Æ&?ûæxNJº~'Øü¯L

1 0

[Glance] Extending Specs Freeze Deadline by 1 week
by Pranali Deore 27 Jun '24

27 Jun '24

Hello All, As discussed in the last glance meeting [1] we are extending the glance specs freeze deadline by 1 week since we have 2 new specs up recently. Current Spec Freeze date (for Glance) : 5th July 2024 New Spec Freeze date (for Glance) : 12th July 2024 Please make sure your specs are updated before the new deadline. [1]: https://meetings.opendev.org/irclogs/%23openstack-meeting/%23openstack-meet… Thanks, Pranali Deore

1 0

[manila] stuck at connecting ceph backend with manila
by Shubham Kumar Yadav 27 Jun '24

27 Jun '24

hello, I'm trying to install manila in my openstack cloud but having problem with "*Authorizing the driver to communicate with Ceph*" *1 -> *i have created a configmap with these confs and mounted it on manila-share pod [client.manila]client mount uid = 0client mount gid = 0log file = /opt/stack/logs/ceph-client.manila.logadmin socket = /opt/stack/status/stack/ceph-$name.$pid.asokkeyring = /etc/ceph/manila.keyring *2 -> *then i have created a secret with client.manila keyring and attached it to the manila-share pod at keyring = /etc/ceph/manila.keyring *3 -> *inside manila-share pod i see the logs 2024-06-03 09:00:53.682 6 INFO manila.share.drivers.cephfs.driver [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] [CEPHFSNATIVE1] Ceph client found, connecting... 2024-06-03 09:00:53.713 6 ERROR manila.share.manager [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Error encountered during initialization of driver CephFSDriver@lab-11053-26006-ceph-2@cephfsnative1: manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 421, in rados_client 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self._rados_client.connect() 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "rados.pyx", line 690, in rados.Rados.connect 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster) 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.713 6 ERROR manila.share.manager During handling of the above exception, another exception occurred: 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/manager.py", line 371, in _driver_setup 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self.driver.do_setup(ctxt) 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 274, in do_setup 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados_client=self.rados_client, 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 424, in rados_client 2024-06-03 09:00:53.713 6 ERROR manila.share.manager raise exception.ShareBackendException( 2024-06-03 09:00:53.713 6 ERROR manila.share.manager manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Finished call to 'manila.share.manager.ShareManager.init_host.<locals>._driver_setup' after 2222.651(s), this was the 12th time calling it. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/after.py:44 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Retrying manila.share.manager.ShareManager.init_host.<locals>._driver_setup in 600.0 seconds as it raised ShareBackendException: [%(be)s] Ceph client failed to connect.. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/before_sleep.py:65 *4 -> *My ceph cluster is healthy 5 -> My manila confs look like (what could be wrong here)? conf: manila: DEFAULT: debug: true enabled_share_protocols: NFS,CIFS,CEPHFS enabled_share_backends: cephfsnative1 cephfsnative1: driver_handles_share_servers: False share_backend_name: CEPHFSNATIVE1 share_driver: manila.share.drivers.cephfs.driver.CephFSDriver cephfs_conf_path: /etc/ceph/ceph.conf cephfs_protocol_helper_type: CEPHFS cephfs_auth_id: manila cephfs_cluster_name: rook-ceph ( my rook-ceph cluster name) cephfs_filesystem_name: myfs (i created this fs manually) *6 -> P*ermissions for the user manila ceph auth get-or-create client.manila -o manila.keyring \ mgr 'allow rw' \ mon 'allow r' *7 -> *ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) openstackhelm Manila Chart name: manila version: 0.1.7 8 -> *ceph -n client.manila fs ls* * [errno 1] RADOS permission error (error connecting to the cluster)* (while it works with admin) Hope to hear from you soon with a solution Thank you!

3 3

[dev][swift][account-reaper] Account with sharded bucket
by Thiago De Moraes Teixeira 27 Jun '24

27 Jun '24

Hi, I'm using swift-account-reaper for accounts with buckets in an unsharded state, and it looks fine, wiping the account. But when I try to wipe an account with sharded buckets, I encounter some problematic scenarios. First, when deleting the account, account-reaper lists objects from the sharded bucket, an empty list is returned from *direct_get_container* function, so the reaper interprets it as an empty bucket, and the delete fails with a 409 Conflict because objects exist inside the bucket. My second thought was to delete the account that holds shard buckets (i.e. .shards_AUTH_<project_id>) so the reaper will wipe this account. Then, with all buckets empty, they'll be deleted from the principal account (AUTH_<project_id>). But in this case, when the reaper tries to delete objects inside this account, all objects are not found because the object path used is "/.shards_AUTH_<project_id>/<bucket>-<hash>- <timestamp>-<shard_index>/<key>" instead of "/AUTH_<project_id>/ <bucket>/<key>". Am I doing something wrong in trying to delete this account? Or does account-reaper, in fact, not support sharded buckets? I'm exploring some patches to make it work, and I can volunteer to work on this fix if necessary. Thanks! -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*

2 5

[neutron] Neutron CI meeting
by Rodolfo Alonso Hernandez 26 Jun '24

26 Jun '24

Hello Neutrinos: This is a personal request. I no longer can attend the CI meeting on Tuesdays at 15:00UTC. Due to the usually low attendance to this meeting, I would like to propose a new slot (if possible) to have this meeting weekly and be able to participate, as I've been doing during the last years. I will also raise this question in the next team meeting (Tuesdays at 14:00UTC). Just out of curiosity, I've created a poll with 3 slots (including the current one), to know what are your preferences. Please, vote only if you usually attend this meeting [1]. Thank you in advance. [1]https://doodle.com/meeting/participate/id/e3KAwKRe

1 0

[nova] Next THREE nova meetings are at risk
by Sylvain Bauza 26 Jun '24

26 Jun '24

Hey folks, Sorry for the very short notice, but I just realized that I can't attend : - today's nova meeting on June 25th (because of some appointment at the same time) - next week's nova meeting on July 2nd (because on PTO for that only day) - other week's nova meeting on July 9th (because of a one-week PTO) As a result, unless someone is proposing to chair one or other meeting, we may need to cancel them. As a reminder, we'll have a spec review day on July 1st, so please prepare your specs in advance. Anyone from the community fancy to run those meetings ? Thanks, -Sylvain

2 1

[openstack-ansible] Centralized logging
by MARCO MOSCATELLI 25 Jun '24

25 Jun '24

HI everyone I'm looking for a system to have centralized logging of all servers and services of an openstack deployment installed via openstack-ansible, have you some suggestions? Elasticsearch or Opensearch? Have you some reference for the installation with openstack-ansible? thanks Best regards Marco

1 0

[neutron] Bug deputy report for week of June 17th
by Brian Haley 25 Jun '24

25 Jun '24

Hi, I was the stand-in Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian Critical bugs ------------- * https://bugs.launchpad.net/neutron/+bug/2069689 - broken out-of-tree modules due to the movement of wsgi module - n-d-r and vpnaas patches have merged - https://review.opendev.org/c/openstack/neutron/+/922185 necessary? High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2069704 - rally jobs broken on unmaintantained/yoga and older branches - https://review.opendev.org/c/openstack/neutron/+/922230 * https://bugs.launchpad.net/neutron/+bug/2069831 - [eventlet] Deprecation of eventlet in Neutron - Rodolfo assigned to himself - All related patches will be tracked here Medium bugs ----------- * https://bugs.launchpad.net/neutron/+bug/2069581 - [WSGI] Missing periodic workers when using WSGI server - https://review.opendev.org/c/openstack/neutron/+/922090 - Other patches still in-progress * https://bugs.launchpad.net/neutron/+bug/2069625 - [OVN] classless-static-route DHCP option is not sanitized and overrides default routes - https://review.opendev.org/c/openstack/neutron/+/922119 * https://bugs.launchpad.net/neutron/+bug/2069718 - [ovn] No connection to VM during live-migration - Seems valid, will discuss at weekly meeting * https://bugs.launchpad.net/neutron/+bug/2069744 - [FT] Run test_periodic_sync_routers_task tests serially - https://review.opendev.org/c/openstack/neutron/+/922218 Low bugs -------- * https://bugs.launchpad.net/neutron/+bug/2069595 - Neutron processes not following the same naming convention - Rodolfo assigned to himself, patches in progress Misc bugs --------- * https://bugs.launchpad.net/neutron/+bug/2069753 - [question] best practices for using RDMA? - Responded with some answers and ask for ML thread Wishlist bugs ------------- * None

1 0

[manila] stuck at connecting ceph backend with manila
by Shubham Kumar Yadav 25 Jun '24

25 Jun '24

hello, I'm trying to install manila in my openstack cloud but having problem *1 -> ceph -n client.manila fs subvolume ls myfs [errno 1] RADOS permission error (error connecting to the cluster)* this is what i get after running that not sure what permissions any idea? i had shared my confs earlier Hope to hear from you soon Thank you!

1 0

Migratekit: near-live migrations from VMware to OpenStack
by Mohammed Naser 25 Jun '24

25 Jun '24

Hi everyone! Over the last few weeks, I’ve been trying to solve the issue of migrations from VMware to OpenStack since they are very downtime heavy at the moment with the existing toolset. In addition, there seems to be no readily available open source projects that can handle the entire process from end to end. After a ton of research and leveraging awesome other open source projects (libnbd, ndbkit, etc), I was able to build a project that allows users to migrate from VMware to OpenStack with almost no downtime. https://github.com/vexxhost/migratekit You can find the documentation on how to use it in the README. I am hoping this is useful for others and that we start to have other people start contributing to the project and testing it in their environments. In all transparency, this was built against an ESXi 8.0 environment (no vCenter) using an evaluation license. We welcome those who have environments that they can test against and hope to increase the coverage for versions. 😊 Please try and give feedback and we hope to that folks find this useful and contribute back to have a fully open sourced way of migrating from VMware. There’s a lot more work to do, but this is a solid start at this point. 😊 Thanks, Mohammed

4 3