October 2020 - openstack-discuss - lists.openstack.org

[nova-scheduler] always the same compute was selected?
by Tony Liu 28 Oct '20

28 Oct '20

Hi, I have a Ussuri setup with 5 compute nodes (same resource spec). When launch 5 Cirros VMs in a batch, they are all launched on the same compute node. All 5 nodes have sufficient resource. AWIK, Nova scheduler picks a compute randomly after going through filtering. But 5 VMs all on the same compute is quite coincidental. Could it be something missing? Is there any control to such random pick? Any way to make a better balancing? >From the below log, could anyone let me know how compute-4 was selected? I tried a few more launches, it's always compute-4 was selected. ========================================= 2020-10-27 13:56:45.484 24 DEBUG nova.scheduler.filter_scheduler [req-e2c6bf4b-7512-4ff0-9899-2a24bffa9131 113ee63a9ed0466794e24d069efc302c 32be191b3ec9497eaec108a22220890b - default default] Filtered [(compute-4, compute-4) ram: 131698MB disk: 14251008MB io_ops: 0 instances: 17, (compute-5, compute-5) ram: 232050MB disk: 14251008MB io_ops: 0 instances: 2, (compute-2, compute-2) ram: 367785MB disk: 14252032MB io_ops: 0 instances: 5, (compute-1, compute-1) ram: 298665MB disk: 14251008MB io_ops: 0 instances: 9, (compute-3, compute-3) ram: 259753MB disk: 14251008MB io_ops: 0 instances: 19] _get_sorted_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:443 2020-10-27 13:56:45.485 24 DEBUG nova.scheduler.filter_scheduler [req-e2c6bf4b-7512-4ff0-9899-2a24bffa9131 113ee63a9ed0466794e24d069efc302c 32be191b3ec9497eaec108a22220890b - default default] Weighed [WeighedHost [host: (compute-4, compute-4) ram: 131698MB disk: 14251008MB io_ops: 0 instances: 17, weight: 2.315526126088031], WeighedHost [host: (compute-2, compute-2) ram: 367785MB disk: 14252032MB io_ops: 0 instances: 5, weight: -999997.0], WeighedHost [host: (compute-1, compute-1) ram: 298665MB disk: 14251008MB io_ops: 0 instances: 9, weight: -999997.2171186721], WeighedHost [host: (compute-3, compute-3) ram: 259753MB disk: 14251008MB io_ops: 0 instances: 19, weight: -999997.3362949106], WeighedHost [host: (compute-5, compute-5) ram: 232050MB disk: 14251008MB io_ops: 0 instances: 2, weight: -999997.371492924]] _get_sorted_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:462 2020-10-27 13:56:45.486 24 DEBUG nova.scheduler.utils [req-e2c6bf4b-7512-4ff0-9899-2a24bffa9131 113ee63a9ed0466794e24d069efc302c 32be191b3ec9497eaec108a22220890b - default default] Attempting to claim resources in the placement API for instance ac10e5df-7e55-4c88-a682-df25cb6dd535 claim_resources /usr/lib/python3.6/site-packages/nova/scheduler/utils.py:1175 2020-10-27 13:56:45.657 24 DEBUG nova.scheduler.filter_scheduler [req-e2c6bf4b-7512-4ff0-9899-2a24bffa9131 113ee63a9ed0466794e24d069efc302c 32be191b3ec9497eaec108a22220890b - default default] [instance: ac10e5df-7e55-4c88-a682-df25cb6dd535] Selected host: (compute-4, compute-4) ram: 131698MB disk: 14251008MB io_ops: 0 instances: 17 _consume_selected_host /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:354 ========================================= Thanks! Tony

2 2

[tc][all] Wallaby Cycle Community Goals
by Graham Hayes 27 Oct '20

27 Oct '20

Hi All It is that time of year / release again - and we need to choose the community goals for Wallaby. Myself and Nate looked over the list of goals [1][2][3], and we are suggesting one of the following: - Finish moving legacy python-*client CLIs to python-openstackclient - Move from oslo.rootwrap to oslo.privsep - Implement the API reference guide changes - All API to provide a /healthcheck URL like Keystone (and others) provide Some of these goals have champions signed up already, but we need to make sure they are still available to do them. If you are interested in helping drive any of the goals, please speak up! We need to select goals in time for the new release cycle - so please reply if there is goals you think should be included in this list, or not included. Next steps after this will be helping people write a proposed goal and then the TC selecting the ones we will pursue during Wallaby. Additionally, we have traditionally selected 2 goals per cycle - however with the people available to do the work across projects Nate and I briefly discussed reducing that to one for this cycle. What does the community think about this? Thanks, Graham 1 - https://etherpad.opendev.org/p/community-goals 2 - https://governance.openstack.org/tc/goals/proposed/index.html 3 - https://etherpad.opendev.org/p/community-w-series-goals 4 - https://governance.openstack.org/tc/goals/index.html#goal-selection-schedule

7 16

[ops][cinder] ceph version survey for operators
by Brian Rosmaita 27 Oct '20

27 Oct '20

Hello Cinder operators, tl;dr - if you use Ceph with your Cinder installation, please fill out this short (4 question) survey before 18:00 UTC on Wednesday 28 October 2020: https://rosmaita.wufoo.com/forms/ceph-version-survey/ Cinder currently contains an RBD driver that supports a Ceph backend, but it does not restrict Ceph to a particular range of versions. We'd like to take advantage of recent developments in Ceph to make some improvements to the RBD driver, but first want to gather data on what versions of Ceph are actually being used with Cinder so we can figure out backward compatibility issues we may face. (Basically, keeping backward compatibility for Ceph versions that don't have the new features will add complexity to the driver code, and more complex code is more likely to have bugs, and nobody likes bugs. Hence, we don't want to carry compatibility code for Ceph versions that no one is using.) If you manage multiple clouds, please fill out a separate survey for each cloud. We plan to discuss this issue at the PTG next week, so please respond before 18:00 UTC on Wednesday 28 October 2020. Here's the survey: https://rosmaita.wufoo.com/forms/ceph-version-survey/ Thank you!

1 1

[PTG] [Diversity] Extra session for Divisive Language discussions
by Amy Marrich 27 Oct '20

27 Oct '20

During the Monday session there were some questions in regards to technical writing and how things should be worded. I have invited Joanne O'Flynn, a Technical Writer from Red Hat to join us on Thursday at 17:00 UTC to help address these questions. This is not a normal PTG time slot but it was the only option due to time zones and already booked sessions for some of the participants from Monday. Thanks, Amy Marrich (spotz)

1 1

[TripleO] [ptg] deployment flow
by Kanevsky, Arkady 27 Oct '20

27 Oct '20

After today's PTG meeting want to make sure I have the right understanding of deployment flow. After deploying undercloud, are these the steps need to be done that are partially outside TripleO (https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/provisi… ): 1. Ironic discovery to discover all nodes. (networking for ironic must be in place for all nodes) 2. Metalsmith will provision baremetal machines - https://docs.openstack.org/metalsmith/latest/ 3. Metalsmith provisions OS on provisioned nodes 4. Something(???) optionally provisions Ceph (based on appropriate roles) 5. Something(???) optionally provisions other 3rd party services, like switch provisioning (based on appropriate roles) 6. Metalsmith optionally provisions Glance (may require Ceph or other 3rd party storage) 7. Metalsmith optionally provisions Neutron (may require switch provisioning) 8. TripleO deploys overcloud Is this the correct flow? Do we still need "openstack overcloud node provision" call or metalsmith already handles it? Or this call uses metalsmith under the cover? Thanks, Arkady

2 3

[requirements] Gerrit breach and commit audit
by Thierry Carrez 27 Oct '20

27 Oct '20

Following the recent gerrit breach, I manually audited all recent changes to the openstack/requirements repository, and could not find suspicious activity. Cheers, -- Thierry Carrez (ttx)

1 0

[cinder] qemu-img info is called as root even if nas_secure_file_operations is set to true
by Jonas Schäfer 27 Oct '20

27 Oct '20

Hi list, We’ve got a curious issue here. We’re trying to configure Cinder for use with NetApp NFS. The NetApp has been configured to disallow superuser access and cinder has been set up with nas_secure_file_operations set to true. While operations like creating volumes work, attaching a volume to an instance fails with the following traceback: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 4369, in _connection_create conn_info = self.driver.initialize_connection(volume, connector) File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/nfs.py", line 134, in initialize_connection volume['name']) File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/nfs.py", line 543, in _qemu_img_info run_as_root=True) File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/remotefs.py", line 765, in _qemu_img_info_base run_as_root=run_as_root) File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 111, in qemu_img_info prlimit=QEMU_IMG_LIMITS) File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 126, in execute return processutils.execute(*cmd, **kwargs) File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 425, in execute cmd=sanitized_cmd) ProcessExecutionError: Unexpected error while running command. Command: /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --force-share /var/lib/cinder/mnt/1be0f17801f5c315c80d99a59518e26e/volume- ae4bba9b-2ab3-4234-8f86-352f1a20e086 Exit code: 1 Stdout: u'' Stderr: u"qemu-img: Could not open '/var/lib/cinder/mnt/ 1be0f17801f5c315c80d99a59518e26e/volume-ae4bba9b-2ab3-4234-8f86-352f1a20e086': Could not open '/var/lib/cinder/mnt/1be0f17801f5c315c80d99a59518e26e/volume- ae4bba9b-2ab3-4234-8f86-352f1a20e086': Permission denied\n" It seems as if cinder executes qemu-img as root. This is confirmed by looking at the code [3]. Looking at the superclass [4], we see that the superclass would’ve decided based on the setting of nas_secure_file_operations if [3] would not pass execute_as_root as True. To us, it makes no sense whatsoever. While [1] mentions something about incorrect permissions on volumes, it does not seem to consider the case of running with NFS Security enabled at all. However, it also seems as if this change went through completely uncontested. Hence, we assume that we are in the wrong here and wanted to ask on the mailing list instead of filing a bug. Does anyone have insight on what we’re doing wrong / missing here? kind regards, Jonas [1]: https://review.opendev.org/#/c/461134/ [2]: https://github.com/openstack/cinder/commit/dbc9c7ca [3]: https://github.com/openstack/cinder/blob/stable/queens/cinder/volume/ drivers/nfs.py#L537-L543 [4]: https://github.com/openstack/cinder/blob/stable/queens/cinder/volume/ drivers/remotefs.py#L753-L761 [5]: https://github.com/openstack/cinder/blob/stable/queens/cinder/volume/ drivers/nfs.py#L431-L432 -- Jonas Schäfer DevOps Engineer Cloud&Heat Technologies GmbH Königsbrücker Straße 96 | 01099 Dresden +49 351 479 367 37 jonas.schaefer(a)cloudandheat.com | www.cloudandheat.com New Service: Managed Kubernetes designed for AI & ML https://managed-kubernetes.cloudandheat.com/ Commercial Register: District Court Dresden Register Number: HRB 30549 VAT ID No.: DE281093504 Managing Director: Nicolas Röhrs Authorized signatory: Dr. Marius Feldmann Authorized signatory: Kristina Rübenkamp

1 0

[masakari] PTG starting Thursday
by Radosław Piliszek 27 Oct '20

27 Oct '20

Hello Everyone! A kind reminder that Masakari has PTG scheduled for the upcoming Thursday and Friday. First session: Thursday October 29, 2020 06:00 - 08:00 (UTC) Feel free to get acquainted with the etherpad [1] in the meantime. [1] https://etherpad.opendev.org/p/masakari-wallaby-vptg Cheers! -yoctozepto

1 0

Overcloud fatal error TASK [Ensure system is NTP time synced]
by dangerzone ar 27 Oct '20

27 Oct '20

Hi, I'm facing overcloud failure at the same line related with 'Ensure system is NTP time synced' *The error below refer:-* *TASK [Ensure system is NTP time synced] ****************************************fatal: [overcloud-controller-0]: FAILED! => {"changed": true, "cmd": ["chronyc", "waitsync", "20"], "delta":......* *controller timedatectl:-* *[heat-admin@overcloud-controller-0 ~]$ timedatectl Local time: Tue 2020-10-27 08:53:19 UTC Universal time: Tue 2020-10-27 08:53:19 UTC RTC time: Tue 2020-10-27 08:53:19 Time zone: UTC (UTC, +0000) NTP enabled: yesNTP synchronized: no RTC in local TZ: no DST active: n/a* *undercloud timedatectl:-* * Local time: Tue 2020-10-27 17:36:11 +08 Universal time: Tue 2020-10-27 09:36:11 UTC RTC time: Tue 2020-10-27 09:36:11 Time zone: Asia/Singapore (+08, +0800) NTP enabled: yesNTP synchronized: yes RTC in local TZ: no DST active: n/a* Tripleo deployment for 1compute and 1controller. I have run few times and end-up with the same error. Seems it's all related to chrony. No more ntp in rocky release and all change to chrony to handle time synchronization. Based on my reading the issue should be resolved in rocky but clearly my overcloud deployment failed at this stage. Please advise how to resolve this problem. Thank you all.

1 0

[cyborg] [ptg] cyborg Wallaby PTG schedule and topics
by yumeng bao 27 Oct '20

27 Oct '20

Hi all, Please check the cyborg PTG schedule below, and here are all the topics[0] that we've proposed, can't wait to see you tomorrow! And we plan to have a team photo on the first day at our internal meeting, come and join! ^^ Tuesday (27th ) 6:00 UTC - 8:00 UTC, Cyborg internal discussion Tuesday (27th) 13:00 - ?, Edge group discussion Wednesday (28th ) 6:00 UTC - 8:00 UTC, Cyborg internal Wednesday (28th): 13:00 - 14:00: Nova-Cyborg cross project session Tuesday (29th ) 6:00 UTC - 8:00 UTC, Cyborg internal [0]https://etherpad.opendev.org/p/cyborg-wallaby-goals Regards， Yumeng

2 1