July 2023 - openstack-discuss - lists.openstack.org

[nova] Scheduler Optimiser
by Dominik Danelski 31 Jul '23

31 Jul '23

Hello, I would like to introduce you to the tool developed under the working title "Scheduler Optimiser". It is meant to test the effectiveness of different Scheduler configurations, both weights and filters on a given list of VM orders and in a semi-realistic infrastructure. My company - CloudFerro - has been preparing in-house for the last few months and foresees publishing the project as FOSS once it reaches the MVP stage. To make the final result more useful to the community and speed up the development (and release), I humbly ask for your expertise: Are you aware of previous similar efforts? Do you notice some flaws in the current approach? What, in your opinion, are more important aspects of the infrastructure behaviour, and what can be relatively safely ignored in terms of the effect on Scheduler results/allocation? Project objectives: * Use Devstack (or another OpenStack deployer) with a real Scheduler to replay a list of compute VM orders, either real from one's infrastructure or artificially created. * Assess the effectiveness of the scheduling in various terms like: "How many machines of a given type can still be allocated at the moment?" using plug-in "success meters". In a strict sense, the project does not simulate THE Scheduler but interacts with it. * Use fake-virt to emulate huge architectures on a relatively tiny test bench. * Have as little as possible, and ideally no changes to the Devstack's code that could not be included in the upstream repository. The usage should be as simple as: 1. Install Devstack. 2. Configure Devstack's cluster with its infrastructure information like flavours and hosts. 3. Configure Scheduler for a new test case. 4. Replay VM orders. 5. Repeat steps 3 and 4 to find better Scheduler settings. * Facilitate creating a minimal required setup of the test bench. Not by replacing standard Devstack scripts, but mainly through tooling for quick rebuilding data like flavours, infrastructure state, and other factors relevant to the simulation. Outside of the scope: * Running continuous analysis on the production environment, even if some plug-ins could be extracted for this purpose. * Retaining information about users and projects when replaying orders. * (Probably / low priority) replaying actions other than VM creation/deletion as they form a minority of operations and ignoring them should not have a distinct effect on the comparison experiments. Current state: Implemented: * Recreating flavours from JSON file exported via OpenStack CLI. * Replaying a list of orders in the form of (creation_date, termination_date, resource_id (optional), flavor_id) with basic flavour properties like VCPU, RAM, and DISK GB. The orders are replayed consecutively. * Plug-in success-rater mechanism which runs rater classes (returning quantified success measure) after each VM add/delete action, retains their intermediate history and "total success" - how it is defined is implementation dependent. First classes interacting with Placement like: "How many VMs of flavours x (with basic parameters for now) can fit in the cluster?" or "How many hosts are empty?". Missing: * Recreating hosts, note the fake-virt remark from "Risks and Challenges". * Tools facilitating Scheduler configuration. * Creating VMs with more parameters like VGPU, traits, and aggregates. * (Lower priority) saving the intermediate state of the cluster during simulation i.e. allocations to analyse it without rerunning the experiment. Currently, only the quantified meters are saved. * Gently failing and saving all information in case of resource depletion: close to completion, handling one exception type in upper layers is needed. * More success meters. Risks and Challenges: * Currently, the tool replays actions one by one, it waits for each creation and deletion to be complete before running success raters and taking another order. Thus, the order of actions is important, but not their absolute time and temporal density. This might skip some side-effects of a realistic execution. * Similarly, to the above, fake-virt provides simple classes that will not reproduce some behaviours of real-world hypervisors. An explicit Scheduler avoids hosts that had recently failed to allocate a VM, but most likely fake-virt will not mock such behaviour. * Fake-virt should reproduce a real diverse infrastructure instead of x copies of the same flavour. This might be the only, but very important change to the OpenStack codebase. If successful, it could benefit other projects and tests as well. Even though the list of missing features is seemingly larger, the most important parts of the program are already there, so we hope to finish the MVP development in a relatively short amount of time. We are going to publish it as FOSS in either case, but as mentioned your observations would be very much welcome at this stage. I am also open to answering more questions about the project. Kind regards Dominik Danelski

3 3

[neutron] bug deputy report for week of 2023-07-24
by Bence Romsics 31 Jul '23

31 Jul '23

Hi All, We had the following new bugs in the last week or so: Unassigned: * https://bugs.launchpad.net/neutron/+bug/2028651 [ovn] IPv6 VIPs broken with ML2/OVN * https://bugs.launchpad.net/neutron/+bug/2028442 Support DNS for ovn_{nb,sb}_connection * https://bugs.launchpad.net/neutron/+bug/2028285 [unit test][xena+] test_port_deletion_prevention fails when runs in isolation Likely valid and high, but my dvr-fu is weak. dvr folks, please have a look: * https://bugs.launchpad.net/neutron/+bug/2028795 Restarting OVS with DVR creates a network loop fix proposed: https://review.opendev.org/c/openstack/neutron/+/889752 High: * https://bugs.launchpad.net/neutron/+bug/2028185 Large number of FIPs causes slow sync_routers response fix proposed: https://review.opendev.org/c/openstack/neutron/+/888915 * https://bugs.launchpad.net/neutron/+bug/2028846 FIP PF don't works with vlan tenant network and ovn backend fix proposed: https://review.opendev.org/c/openstack/neutron/+/889871 Medium: * https://bugs.launchpad.net/neutron/+bug/2028161 [ovn-octavia-provider] FIP not included into LogicalSwitchPortUpdate event handler method fix merged on master, backports proposed: https://review.opendev.org/q/Ibee3906e8e9575fba7811e989e3e111a026ce45b Low: * https://bugs.launchpad.net/neutron/+bug/2028724 neutron should forbid configuring agent_down_time that is known to crash due to CPython epoll limitation fix proposed: https://review.opendev.org/c/openstack/neutron/+/889373 RFE: * https://bugs.launchpad.net/neutron/+bug/2028660 [fwaas][rfe] support list type of port_range for firewall rule Incomplete: * https://bugs.launchpad.net/neutron/+bug/2028338 Router with l3_agent state unknown Invalid: * https://bugs.launchpad.net/neutron/+bug/2028544 dhcp agent binding count greather than dhcp_agents_per_network Cheers, Bence (rubasov)

1 0

[OSA] Failover at the storage level
by Murilo Morais 29 Jul '23

29 Jul '23

Good morning everybody! I'm trying to raise a scenario with redundancy at the storage level. I'm using CEPH with RBD-mirror. Cinder has a pretty clear documentation about it, but I didn't find anything about Glance and the "nova_libvirt_images_rbd_pool" and the "cinder_service_backup_driver". What would be the best approach? Only use Cinder with Mirror? If by chance the "nova_libvirt_images_rbd_pool" dies, if you are using config-drive the disk will die, correct? In this case, would it be better to leave local? I can't see what would be the best scenario for Failover cases with OSA. Thanks in advance!

2 1

[openstack-helm] [ceph deployment failed] error:- MountVolume.SetUp failed for volume "ceph-client-admin-keyring" secret "ceph-client-admin-keyring" not found
by Viswanath Ediga 29 Jul '23

29 Jul '23

Team, I am trying to deploy openstack using OSH. I am following documentation https://docs.openstack.org/openstack-helm/latest/install/multinode.html I am getting error while "Deploying ceph" phase. container creation is failing. Below is the error that I see in POD description Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 24m (x73 over 3h7m) kubelet Unable to attach or mount volumes: unmounted volumes=[ceph-client-admin-keyring], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition Warning FailedMount 4m18s (x99 over 3h9m) kubelet MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secret "ceph-client-admin-keyring" not found Regards, Any help is appreciated. Kasi Viswanath

1 0

[manila] Collaborative Review session on resource locks
by Goutham Pacha Ravi 29 Jul '23

29 Jul '23

Hello Zorillas and other interested stackers, We're hosting a collab-review session on the changes pertaining to resource locks [1]. These changes are being pursued for the 2023.2/Bobcat release. The collab-review meeting will be at 1530 UTC on Thursday, Aug 3rd 2023 (gcal [2]). The meeting link will be published on the etherpad closer to the event, and shared during the IRC meeting that begins at 1500 UTC on the same day. [3] A reminder for the unfamiliar, a "collab-review" is a regular activity that the team does where patch authors provide a walkthrough of the code changes on a virtual community meeting and answer questions real time. This has helped us speed up reviews in past cycles. While the intention is to brainstorm synchronously, we will reflect all feedback - big and small, on Gerrit for posterity. In addition, the meeting call will be recorded, and the recording will be made available on the community's Youtube channel. [4] Hoping to see you there! Goutham [1] https://etherpad.opendev.org/p/manila-allow-locking-shares-against-deletion [2] https://red.ht/manila-bobcat-collab-review [3] https://wiki.openstack.org/wiki/Manila/Meetings#Next_Meeting [4] https://www.youtube.com/@openstackmanila

1 0

[charms][sunbeam] Proposing Guillaume Boutry for charms-core
by James Page 28 Jul '23

28 Jul '23

Hi All Guillaume has been working on the Sunbeam Charms for the last few months and has proven an able reviewer and contributor to the charms; we need to dis-entangle the sunbeam project charms from the main openstack charms set but until that work is completed I'd like to propose Guillaume as a member of the charms-core team. Cheers James

3 3

[nova] Nova meetings on Aug 1 and Aug 8 are CANCELLED
by Sylvain Bauza 28 Jul '23

28 Jul '23

As said above, we don't have quorum. Gibi will be chairing on Aug 15 1600UTC. As a reminder, any person is welcome (even if you are not a contributor) during our IRC meetings. You can find all our weekly minutes and our connection details in https://meetings.opendev.org/#Nova_Team_Meeting Thanks, -Sylvain

1 0

[cloudkitty] question to the scope_key in cloudkitty
by Noah Elias Feldt 28 Jul '23

28 Jul '23

Hello! I am currently trying to get CloudKitty to work with the Prometheus Collector and PyScripts. I noticed that the scope_key for Prometheus can only be changed once in the global configuration file. This was a problem for me because we need different scope_keys for different metrics. Many of the metrics still need the scope key "tenant_id" and some others "project_id". I have found in the documentation of CloudKitty no way to change this for each metric, so I have internally changed something in the source code and added the function to set the scope_key for each metric individually. I just wanted to ask if you would be interested if I submit the small change upstream? Thank you. Regards Noah Feldt

3 3

[Octavia] Weekly meetings cancelled
by Gregory Thiemonge 28 Jul '23

28 Jul '23

Hi, As discussed during the weekly meeting, the next 2 Octavia meetings are cancelled. The next meeting will be on Aug 16. Thanks Greg

1 0

[requirements] Blockdiag replacement (not working with Pillow 10)
by Karolina Kula 27 Jul '23

27 Jul '23

Hello, There are projects[1] which are depending on python-sphinxcontrib-blockdiag[2], which depend on blockdiag [3]. Blockdiag project has no recent changes and seems to be unmaintained. Current version of blockdiag is not compatible with Pillow 10 [4] which is included in current in-flight requirements update [5]. I was trying to contact the maintainer, but with no success. What'd be the best way to proceed? Are there any replacements for this project? [1] https://codesearch.opendev.org/?q=blockdiag&i=nope&literal=nope&files=requi… [2] https://github.com/blockdiag/sphinxcontrib-blockdiag [3] https://github.com/blockdiag/blockdiag [4] https://github.com/blockdiag/blockdiag/pull/174 [5] https://review.opendev.org/c/openstack/requirements/+/884564 Regards, Karolina Kula

3 2