July 2022 - openstack-discuss - lists.openstack.org

RAM and Storage requirements for Openstack cloud
by Mahendra Paipuri 27 Jul '22

27 Jul '22

Hello all, We are going to deploy Openstack cloud with researchers as primary target users. We will have around 100 GPUs with 12-15 servers with Infiniband interconnect. We still do not know the exact spec of servers nor GPUs but mostly we will have A100s and Intel Xeon processors. What sort of RAM and Storage requirements we need for a cluster of this size? Of course, this depends a lot on use cases and this cloud will be primarily used for HPC and AI. For the storage, we are mainly interested in the block storage requirements for provisioning VMs. We will most probably have a shared parallel file system as scratch and project spaces. Is there any rule of thumb to get to the RAM and storage requirement numbers based on compute infrastructure we will have? If we can estimate a sort of "lower bound" that would be really helpful for us. If anyone have clusters of this size at your organizations and if you can share the RAM ans storage details of your clouds, that would be very useful for us too. Thanks a lot and have a great day!! Regards Mahendra

3 3

Customization of scheduler manager
by Levon Melikbekjan 27 Jul '22

27 Jul '22

Hi all, as part of my thesis, I modified the Openstack version Train with the intention of sharing resources, which are not in use with other users to ensure maximum utilization. So far everything is working fine except for the last step. Let me first explain my work. The system works according to the following rules: 1. Users who own compute hosts within the private cloud have the highest priority on their hosts. 2. Users who do not own hosts within the private cloud are low priority users who can instantiate their virtual machine on unused resources (on the hosts that have an owner). 3. If the owner wants to use his resources that are currently occupied, the foreign VM must be suspended to free up resources for the owners VM. 4. An owner is a low priority user on foreign hosts. Everything works automatically and generically, but in step 3 I do not suspend those VMs, I delete them. I want the VMs to be suspended to be able to restart them with the intention of being able to continue processes that are paused, and I know there is maybe a REST API functions that provides this functionality. A user should be able to continue his work after resources become free again. It would be annoying if long-running processes were killed. My question is this: Is the suspend function the right choice? Are the resources released when I use the suspend function? Thank you & Best regards, Levon Melikbekjan

2 2

[dev][horizon][keystone]
by Sergey Drozdov 27 Jul '22

27 Jul '22

To whom it may concern, I have previously sen out an email about this topic but just wanted to run this by community again incase it was missed. I have the following issue. The firm I am currently working at is running OpenStack at sale with circa 6000 different projects. Whenever we try to access a projects tab via horizon we experience a timeout (accessing through the API works fine). For the horizon team, we were hoping to propose something akin to filtering/dynamic listing in order to rectify the issue. For the keystone team, we were wondering whether there is anything we can do with the API in order to simplify the aforementioned horizon proposal; we are hoping to get both teams on board. We are ready to open up a bug report and start subsequent development should this be supported by both teams! I am available on the both Openstack-horizon and Openstack-keystone irc channels under the nickname sdrozdov, please feel free to reach out. Best Regards, Sergey Drozdov Software Engineer The Hut Group

3 2

[cinder] Bug deputy report for week of 07-27-2022
by Sofia Enriquez 27 Jul '22

27 Jul '22

This is a bug report from 07-20-2022 to 07-27-2022. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/cinder/+bug/1982436 "Reimage volume API image is invalid, the status should not be downloading." Fix proposed to master. - https://bugs.launchpad.net/cinder/+bug/1982848 "Creating from source volume tries to create it multiple times if rekeying fails." Unassigned. Low - https://bugs.launchpad.net/cinder/+bug/1982350 "Infinidat Cinder driver multi-attach feature is broken." Fix proposed to master. - https://bugs.launchpad.net/cinder/+bug/1982405 "Infinidat Cinder driver generic volume migration feature is broken." Fix proposed to master. - https://bugs.launchpad.net/cinder/+bug/1982568 "PowerMax doesn´t work with the workload in extra specs." Unassigned. - https://bugs.launchpad.net/cinder/+bug/1982891 "NFS Attach Encrypted fails only on the first attempt." Unassigned. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[cinder] This week's meeting will be in video+IRC
by Rajat Dhasmana 27 Jul '22

27 Jul '22

Hello Argonauts, This week's meeting (today) will be held in video + IRC mode with details as follows: Date: 27th July, 2022 Time: 1400 UTC Meeting link: https://bluejeans.com/556681290 IRC Channel: #openstack-meeting-alt Make sure you're connected to both the bluejeans meeting and IRC since we do roll call and also (sometimes) summarize the discussion points on IRC. Thanks and regards Rajat Dhasmana

1 0

[Kolla][14.1.0][Yoga][Fluentd] fluentd container restarting indefinitely
by Vishwanath 26 Jul '22

26 Jul '22

Hello all, I have upgraded openstack, we are currently running on 14.1.0. I noticed fluentd container restarting indefinitely. The message I see under /var/log/kolla/fluentd/fluentd.log is as follows, Any thoughts on how to fix this ? I noticed a similar issue back in 2017 from this post - https://bugs.launchpad.net/kolla-ansible/+bug/1663126 *error log:* *2022-07-06 21:20:42 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="'format' parameter is required"* *full logs:* 2022-07-06 21:20:42 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf" 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.3' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.1.1' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-grep' version '0.3.4' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-kafka' version '0.14.1' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-parser' version '0.6.1' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.8.2' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.0.2' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-s3' version '1.4.0' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-td' version '1.1.0' 2022-07-06 21:20:42 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.5' 2022-07-06 21:20:42 +0000 [info]: gem 'fluentd' version '1.11.2' 2022-07-06 21:20:42 +0000 [info]: gem 'fluentd' version '0.12.43' 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf97620 @keys="programname">, /^(cinder-api-access|cloudkitty-api-access|gnocchi-api-access|horizon-access|keystone-apache-admin-access|keystone-apache-public-access|monasca-api-access|octavia-api-access|placement-api-access)$/, "", "apache_access", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf96f90 @keys="programname">, /^(aodh_wsgi_access|barbican_api_uwsgi_access|zun_api_wsgi_access|vitrage_wsgi_access)$/, "", "wsgi_access", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf96950 @keys="programname">, /^(nova-api|nova-compute|nova-compute-ironic|nova-conductor|nova-manage|nova-novncproxy|nova-scheduler|nova-placement-api|placement-api|privsep-helper)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf96310 @keys="programname">, /^(sahara-api|sahara-engine)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf95cd0 @keys="programname">, /^(neutron-server|neutron-openvswitch-agent|neutron-ns-metadata-proxy|neutron-metadata-agent|neutron-l3-agent|neutron-dhcp-agent)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf95690 @keys="programname">, /^(magnum-conductor|magnum-api)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf95050 @keys="programname">, /^(keystone)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf94a10 @keys="programname">, /^(heat-engine|heat-api|heat-api-cfn)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf943d0 @keys="programname">, /^(glance-api)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabf850 @keys="programname">, /^(cloudkitty-storage-init|cloudkitty-processor|cloudkitty-dbsync|cloudkitty-api)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabe978 @keys="programname">, /^(ceilometer-polling|ceilometer-agent-notification)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabdcd0 @keys="programname">, /^(barbican-api|barbican-worker|barbican-keystone-listener|barbican-db-manage|app)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adabc4e8 @keys="programname">, /^(aodh-notifier|aodh-listener|aodh-evaluator|aodh-dbsync)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9fa50 @keys="programname">, /^(cyborg-api|cyborg-conductor|cyborg-agent)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9f410 @keys="programname">, /^(cinder-api|cinder-scheduler|cinder-manage|cinder-volume|cinder-backup|privsep-helper)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9edd0 @keys="programname">, /^(mistral-server|mistral-engine|mistral-executor)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9e790 @keys="programname">, /^(designate-api|designate-central|designate-manage|designate-mdns|designate-sink|designate-worker)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9e150 @keys="programname">, /^(manila-api|manila-data|manila-manage|manila-share|manila-scheduler)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9db10 @keys="programname">, /^(trove-api|trove-conductor|trove-manage|trove-taskmanager)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9d4d0 @keys="programname">, /^(murano-api|murano-engine)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9ce90 @keys="programname">, /^(senlin-api|senlin-conductor|senlin-engine|senlin-health-manager)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9c850 @keys="programname">, /^(watcher-api|watcher-applier|watcher-db-manage|watcher-decision-engine)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adf9c210 @keys="programname">, /^(freezer-api|freezer-api_access|freezer-manage)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adab7880 @keys="programname">, /^(octavia-api|octavia-health-manager|octavia-housekeeping|octavia-worker)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adab6ef8 @keys="programname">, /^(zun-api|zun-compute|zun-cni-daemon)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adab5918 @keys="programname">, /^(kuryr-server)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfabe90 @keys="programname">, /^(gnocchi-api|gnocchi-statsd|gnocchi-metricd|gnocchi-upgrade)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfab850 @keys="programname">, /^(ironic-api|ironic-conductor|ironic-inspector)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfab210 @keys="programname">, /^(tacker-server|tacker-conductor)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfaabd0 @keys="programname">, /^(vitrage-ml|vitrage-notifier|vitrage-graph|vitrage-persistor)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfaa590 @keys="programname">, /^(blazar-api|blazar-manager)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfa9f50 @keys="programname">, /^(monasca-api|monasca-notification|monasca-persister|agent-collector|agent-forwarder|agent-statsd)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfa9910 @keys="programname">, /^(masakari-engine|masakari-api)$/, "", "openstack_python", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: programname [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfa92d0 @keys="programname">, /.+/, "", "unmatched", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: Payload [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfc9cb0 @keys="Payload">, /^\d{6}/, "", "infra.mariadb.mysqld_safe", nil] 2022-07-06 21:20:42 +0000 [info]: adding rewrite_tag_filter rule: Payload [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00005634adfc9670 @keys="Payload">, /^\d{4}-\d{2}-\d{2}/, "", "infra.mariadb.mysqld", nil] *2022-07-06 21:20:42 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="'format' parameter is required"* Thanks Vish

3 4

[nova] hw_qemu_guest_agent image metadata property
by John Ratliff 26 Jul '22

26 Jul '22

We forgot to set this property on our image before creating several servers in our openstack cluster. I tried adding this property to the volume of one of the hosts and shutting it down / turning it back on, but it didn’t help. I found this blog post showing one way that worked, but it requires manual database editing. I was hoping for something a bit less typo-prone, like something in the openstack cli. https://www.thenoccave.com/2021/07/openstack-qemu-guest-tools-after-creatio… I can automate the above with ansible if that’s the only way, but if someone has a better suggestion that doesn’t involve recreating our instances, I would appreciate it. Thanks. --John Ratliff

3 3

[dev][security-sig][tc] Please follow up on privately reported defects
by Jeremy Stanley 26 Jul '22

26 Jul '22

First, a huge thank you to everyone who is staying on top of reports of suspected security vulnerabilities! Unfortunately, not everyone has been, which is the reason for this E-mail. It's common practice that, if someone finds a problem in software which they think might be an exploitable security vulnerability, they report it initially in private in order to give the project's maintainers an opportunity to correct things and have patches ready before it becomes common knowledge. This works okay as long as people actually look at these privately reported bugs (or at the project's bugs at all). For OpenStack deliverables whose maintainers opt them into VMT oversight[*], these private reports are initially handled by a vulnerability coordinator in order to make sure that they're probably reported against the correct project, that the project maintainers who have volunteered to handle those sorts of reports are correctly subscribed, and that everyone is reminded of the ground rules and timetable for resolving reports under such an embargo. For other OpenStack deliverables, VMT members may still weigh in on those private reports and offer assistance or guidance on handling and reporting procedures. Our VMT members do not, however, have sufficient time in their day to keep individually reaching out to project maintainers in order to remind them to do their part. OpenStack is a community which has optimized around transparency and public collaboration, so it's not surprising that confirming bugs and reviewing changes in private is clunky and unpleasant. This is, if anything, a reason to prioritize triaging private bug reports in order to make sure they're really a bug (not just a misunderstanding or misconfiguration), and represent a severe enough risk to warrant continued handling in secret. Many of the private bug reports currently pending could probably be switched to public and even perhaps closed today, if maintainers for their projects would just find a moment to take a look at them. For the ones which can't be handled right away, at least leave a quick comment letting the reporter and the VMT members know you're taking a look, or any first impressions or questions you might have. If you're interested in helping a project resolve reported vulnerabilities and aren't yet a member of their security review team in the appropriate bug tracker (usually *-coresec in LP or openstack-security-* in SB), then please reach out to the appropriate PTL and let them know. If you're a PTL and you were never made a member of the security review team for your project or are having trouble adding willing volunteers, please follow up here on the ML or feel free to reach out to me directly for assistance. For those who read this far, thank you for your time, and please remember to follow up on those bugs! [*] https://security.openstack.org/repos-overseen.html -- Jeremy Stanley

1 0

[nova] Hold your rechecks
by Sylvain Bauza 26 Jul '22

26 Jul '22

We're currently investigating a quite impacting issue in both nova-next and ovs-hybrid-plug jobs that prevent Zuul to be happy. https://bugs.launchpad.net/nova/+bug/1940425 In the meantime, as the failure rate is pretty high, doing rechecks (against this bug number of course, you already know to NOT do blind rechecks [1]) doen't help and just consumes constrained CI resources for a very little and unpredictable gain. Thanks, -Sylvain

2 3

[kolla] RabbitMQ High Availability
by Tan Tran Trong 26 Jul '22

26 Jul '22

Hello, I'm trying to figure out how to configure RabbitMQ to make it high available. I have 3 controller nodes and 2 compute nodes, deployed with kolla with mostly default configuration. The RabbitMQ set to ha-all for all queues on all nodes, amqp_durable_queues = True My problem is when I shutdown 1 controller node (or 1 RabbitMQ container) (master or slave) the whole cluster becomes unstable. Some instances can not be created, it is stuck on Scheduling, Block Device Mapping, the volumes not shown or are stuck on creating, the compute node reported dead randomly,... I'm looking for documentation to know how Openstack using RabbitMQ, Openstack behavior when RabbitMQ node down and way to make RabbitMQ HA in a stable way. Do you have any recommendation? TIA, Tan

5 6