- openstack-discuss - lists.openstack.org

[placement][sdk] How to debug HTTP 502 errors with placement in DevStack?
by Stephen Finucane 03 Aug '23

03 Aug '23

We recently merged support for placement traits in openstacksdk. Since then, we've seen an uptick in failures of various functional jobs [1]. The failure is always the same test: openstack.tests.functional.placement.v1.test_trait.TestTrait.test_resource_pr ovider_inventory That test simply creates a new, custom trait and then attempts to list all traits, show an individual trait, and finally delete the trait. The failure occurs during the first step, creation of the custom trait: openstack.exceptions.HttpException: HttpException: 502: Server Error for url: https://10.209.100.9/placement/traits/CUSTOM_A982E0BA1C2B4D08BFD6D2594C6783… , Bad Gateway: response from an upstream server.: The proxy server received an invalid: Apache/2.4.52 (Ubuntu) Server at 10.209.100.9 Port 80: Additionally, a 201 Created: 502 Bad Gateway: error was encountered while trying to use an ErrorDocument to handle the request. I've looked through the various job artefacts and haven't found any smoking guns. I can see placement receive and reply to the request so it would seem something is happening in between. *Fortunately*, this is also reproducible locally against a standard devstack deployment by running the following in the openstacksdk repo: OS_TEST_TIMEOUT=60 tox -e functional-py310 -- \ -n openstack/tests/functional/placement/v1/test_trait.py \ --until-failure Does anyone have any insight into what could be causing this issue and have suggestions for how we might go about debugging it? As things I haven't a clue 😔 Cheers, Stephen [1] https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk-functiona…

2 2

[magnum][openstack-ansible][k8s] kube_masters CREATE_FAILED
by Satish Patel 03 Aug '23

03 Aug '23

Folks, I am running the Xena release and fedora-coreos-31.X image. My cluster is always throwing an error kube_masters CREATE_FAILED. This is my template: openstack coe cluster template create --coe kubernetes --image "fedora-coreos-35.20220116" --flavor gen.medium --master-flavor gen.medium --docker-storage-driver overlay2 --keypair jmp1-key --external-network net_eng_vlan_39 --network-driver flannel --dns-nameserver 8.8.8.8 --labels="container_runtime=containerd,cinder_csi_enabled=false" --labels kube_tag=v1.21.11-rancher1,hyperkube_prefix=docker.io/rancher/ k8s-new-template-31 Command to create cluster: openstack coe cluster create --cluster-template k8s-new-template-31 --master-count 1 --node-count 2 --keypair jmp1-key mycluster31 Here is the output of heat stack [root@ostack-eng-osa images]# heat resource-list mycluster31-bw5yi3lzkw45 WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead +-------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------+-----------------+----------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | +-------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------+-----------------+----------------------+ | api_address_floating_switch | | Magnum::FloatingIPAddressSwitcher | INIT_COMPLETE | 2023-08-01T20:55:49Z | | api_address_lb_switch | | Magnum::ApiGatewaySwitcher | INIT_COMPLETE | 2023-08-01T20:55:49Z | | api_lb | 99e0f887-fbe2-4b2f-b3a1-b1834c9a21c2 | file:///openstack/venvs/magnum-24.5.1/lib/python3.8/site-packages/magnum/drivers/common/templates/lb_api.yaml | CREATE_COMPLETE | 2023-08-01T20:55:49Z | | etcd_address_lb_switch | | Magnum::ApiGatewaySwitcher | INIT_COMPLETE | 2023-08-01T20:55:49Z | | etcd_lb | d4ba15f3-8862-4f2b-a2cf-53eafd36d286 | file:///openstack/venvs/magnum-24.5.1/lib/python3.8/site-packages/magnum/drivers/common/templates/lb_etcd.yaml | CREATE_COMPLETE | 2023-08-01T20:55:49Z | | kube_cluster_config | | OS::Heat::SoftwareConfig | INIT_COMPLETE | 2023-08-01T20:55:49Z | | kube_cluster_deploy | | OS::Heat::SoftwareDeployment | INIT_COMPLETE | 2023-08-01T20:55:49Z | | kube_masters | 9ac8fc3e-a7d8-4eca-90c6-f66a8e0c43f0 | OS::Heat::ResourceGroup | CREATE_FAILED | 2023-08-01T20:55:49Z | | kube_minions | | OS::Heat::ResourceGroup | INIT_COMPLETE | 2023-08-01T20:55:49Z | | master_nodes_server_group | 19c9b300-f655-4db4-b03e-ea1479c541db | OS::Nova::ServerGroup | CREATE_COMPLETE | 2023-08-01T20:55:49Z | | network | a908f229-fe8f-4ab8-b245-e8cf90c1b233 | file:///openstack/venvs/magnum-24.5.1/lib/python3.8/site-packages/magnum/drivers/common/templates/network.yaml | CREATE_COMPLETE | 2023-08-01T20:55:49Z | | secgroup_kube_master | 79e6b233-1a18-48c4-8a4f-766819eb945f | OS::Neutron::SecurityGroup | CREATE_COMPLETE | 2023-08-01T20:55:49Z | | secgroup_kube_minion | 2a908ffb-15bf-45c5-adad-6930b0313e94 | OS::Neutron::SecurityGroup | CREATE_COMPLETE | 2023-08-01T20:55:49Z | | secgroup_rule_tcp_kube_minion | 95779e79-a8bc-4ed4-b035-fc21758bd241 | OS::Neutron::SecurityGroupRule | CREATE_COMPLETE | 2023-08-01T20:55:49Z | | secgroup_rule_udp_kube_minion | 2a630b3e-51ca-4504-9013-353cbe7c581b | OS::Neutron::SecurityGroupRule | CREATE_COMPLETE | 2023-08-01T20:55:49Z | | worker_nodes_server_group | d14b0630-95fa-46dc-81e3-2f90e62c7943 | OS::Nova::ServerGroup | CREATE_COMPLETE | 2023-08-01T20:55:49Z | +-------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------+-----------------+----------------------+ I can ssh into an instance but am not sure what logs I should be chasing to find the proper issue. Any kind of help appreciated

3 5

[cloudkitty] InfluxDB version 1.8.10
by Kaster, Jörn 03 Aug '23

03 Aug '23

Hello together, we have deployed OpenStack with the standard kolla-ansible toolset. Within is included the rating component cloudkitty with prometheus and InfluxDB. The InfluxDB version that is deployed with kolla-ansible is 1.8.10 [2021-10-11]. The actual newest InfluxDB version is 2.7.1 [2023-04-28]. Are there any plans to migrate in the near future to the newest version? I couldn't find any informations on InfluxDB Sites if the 1.8 branch is supported anymore. It seems not like described in Point 2.4 of [1]. [1] https://www.influxdata.com/legal/support-policy/ [https://images.ctfassets.net/o7xu9whrs0u9/fn7Q8NJ8ctkA2FOf8DPjW/e2c782d7edb…]<https://www.influxdata.com/legal/support-policy/> Support Policy | InfluxData<https://www.influxdata.com/legal/support-policy/> InfluxData Support Program describes InfluxData’s current support offerings and support policies for the Software and Subscription Services. www.influxdata.com

2 3

[ptl][osc][sdk] openstackclient/sdk "service cores"
by Brian Rosmaita 02 Aug '23

02 Aug '23

Hello Stackers, At the Vancouver 2023 Forum, Artem and Stephen proposed adding individual OpenStack project cores as "service cores" for the python-openstackclient and openstacksdk projects.[0] This will give -2..+2 review powers to subject matter experts for osc/sdk changes affecting particular openstack components, and hopefully speed up the reviewing process. The idea is that service-cores will verify that the cli/sdk code is correct for their particular openstack component, and the "regular" osc/sdk core team (which will have sole +W power) will ensure that changes preserve consistency across the entire osc/sdk. You can give your core team permissions to act as osc/sdk "service cores" by proposing a patch to project-config. You can use the following patch as an example: https://review.opendev.org/c/openstack/project-config/+/890346 cheers, brian [0] https://etherpad.opendev.org/p/oscsdk-vancouver-forum-2023

1 0

[SDK][OSC] Core team cleanup
by Stephen Finucane 02 Aug '23

02 Aug '23

👋️ The openstacksdk-core Gerrit team contains a number of individuals who have not contributed to SDK in at over a year. * Adrian Turjak * Dean Troyer * Doug Hellmann * Monty Taylor * Paul Belanger In addition, the python-openstackclient-core Gerrit group contains the following individuals who have not contributed to OSC in the same time period. * Dean Troyer * Doug Hellmann * Matt Riedemann * Monty Taylor So that I don't have to remember to follow-up on this in a week's time, I have gone ahead and preemptively removed these individuals from the respective groups. If you are on either list and wish to remain in the groups, please reach out to me and I'll re-add you promptly :) Alternatively, assuming these folks are indeed no longer working on SDK or OpenStack at large, thank you all for your historical contributions. As is customary with emeritus reviewers, if at a future date you wish to resume OpenStack contributions we'd be happy to re-add you to the openstacksdk-core group. Thanks, Stephen

2 1

Cinder Bug Report 2023-08-02
by Rajat Dhasmana 02 Aug '23

02 Aug '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Low - RBD migrate_volume code optimization <https://bugs.launchpad.net/cinder/+bug/2028793> - Status: Recently reported, no comments or fix proposed - NetApp ONTAP - Error on failover-host with REST API <https://bugs.launchpad.net/cinder/+bug/2028857> - Status: Fix proposed <https://review.opendev.org/c/openstack/cinder/+/889826> - [Pure Storage] failure to disconnect remote hosts in uniform sync rep cluster <https://bugs.launchpad.net/cinder/+bug/2029005> - Status: No comments or fix proposed Thanks Rajat Dhasmana

1 0

[openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate
by Nguyễn Hữu Khôi 02 Aug '23

02 Aug '23

Hello guys. I am using openstack yoga with kolla ansible. When I migrate: instance1 from host A to host B after that I cannot ping this instance(telnet also). I must restart neutron_openvswitch_agent or move this instance back to host B then this problem has gone. this is my settings: ----------------- neutron.conf ----------------- [nova] live_migration_events = True ------------------------------------------------ ----------------- nova.conf ----------------- [DEFAULT] vif_plugging_timeout = 600 vif_plugging_is_fatal = False debug = True [compute] live_migration_wait_for_vif_plug = True [workarounds] enable_qemu_monitor_announce_self = True ----------------- openvswitch_agent.ini----------------- [securitygroup] firewall_driver = openvswitch [ovs] openflow_processed_per_port = true I check nova, neutron, ops logs but they are ok. Thank you. Nguyen Huu Khoi

4 12

[tc] Technical Committee next weekly meeting today on August 1, 2023
by Nikolla, Kristi 01 Aug '23

01 Aug '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held today (Tuesday, Aug 1, 2023) at 1800 UTC on Zoom. Use the following link to connect https://us06web.zoom.us/j/87108541765?pwd=emlXVXg4QUxrUTlLNDZ2TTllWUM3Zz09 The agenda for the meeting is: • Roll call • Follow up on past action items • tc-members to review https://review.opendev.org/c/openstack/project-team-guide/+/843457 • Unmaintained status replaces Extended Maintenance • https://review.opendev.org/c/openstack/governance/+/888771 • Release notes guidelines for SLURP/NON-SLURP cadence • Gate health check • Open Discussion and Reviews • https://review.opendev.org/q/projects:openstack/governance+is:open Thank you, Kristi Nikolla

2 2

Tap as a service on openstack with Juju and MaaS
by Jean-Bernard Altidor 31 Jul '23

31 Jul '23

Hello, I've been struggling to configure tap as a service on openstack for a few months now. I can't find any guide or anything about it. Can you suggest a method of port mirroring? I've even via the ovs bridge but I really need a way to mirror data and send it to an instance for analysis as I'm working on an IDS. Thanks in advance for any ideas ! JB

2 2

[nova] Scheduler Optimiser
by Dominik Danelski 31 Jul '23

31 Jul '23

Hello, I would like to introduce you to the tool developed under the working title "Scheduler Optimiser". It is meant to test the effectiveness of different Scheduler configurations, both weights and filters on a given list of VM orders and in a semi-realistic infrastructure. My company - CloudFerro - has been preparing in-house for the last few months and foresees publishing the project as FOSS once it reaches the MVP stage. To make the final result more useful to the community and speed up the development (and release), I humbly ask for your expertise: Are you aware of previous similar efforts? Do you notice some flaws in the current approach? What, in your opinion, are more important aspects of the infrastructure behaviour, and what can be relatively safely ignored in terms of the effect on Scheduler results/allocation? Project objectives: * Use Devstack (or another OpenStack deployer) with a real Scheduler to replay a list of compute VM orders, either real from one's infrastructure or artificially created. * Assess the effectiveness of the scheduling in various terms like: "How many machines of a given type can still be allocated at the moment?" using plug-in "success meters". In a strict sense, the project does not simulate THE Scheduler but interacts with it. * Use fake-virt to emulate huge architectures on a relatively tiny test bench. * Have as little as possible, and ideally no changes to the Devstack's code that could not be included in the upstream repository. The usage should be as simple as: 1. Install Devstack. 2. Configure Devstack's cluster with its infrastructure information like flavours and hosts. 3. Configure Scheduler for a new test case. 4. Replay VM orders. 5. Repeat steps 3 and 4 to find better Scheduler settings. * Facilitate creating a minimal required setup of the test bench. Not by replacing standard Devstack scripts, but mainly through tooling for quick rebuilding data like flavours, infrastructure state, and other factors relevant to the simulation. Outside of the scope: * Running continuous analysis on the production environment, even if some plug-ins could be extracted for this purpose. * Retaining information about users and projects when replaying orders. * (Probably / low priority) replaying actions other than VM creation/deletion as they form a minority of operations and ignoring them should not have a distinct effect on the comparison experiments. Current state: Implemented: * Recreating flavours from JSON file exported via OpenStack CLI. * Replaying a list of orders in the form of (creation_date, termination_date, resource_id (optional), flavor_id) with basic flavour properties like VCPU, RAM, and DISK GB. The orders are replayed consecutively. * Plug-in success-rater mechanism which runs rater classes (returning quantified success measure) after each VM add/delete action, retains their intermediate history and "total success" - how it is defined is implementation dependent. First classes interacting with Placement like: "How many VMs of flavours x (with basic parameters for now) can fit in the cluster?" or "How many hosts are empty?". Missing: * Recreating hosts, note the fake-virt remark from "Risks and Challenges". * Tools facilitating Scheduler configuration. * Creating VMs with more parameters like VGPU, traits, and aggregates. * (Lower priority) saving the intermediate state of the cluster during simulation i.e. allocations to analyse it without rerunning the experiment. Currently, only the quantified meters are saved. * Gently failing and saving all information in case of resource depletion: close to completion, handling one exception type in upper layers is needed. * More success meters. Risks and Challenges: * Currently, the tool replays actions one by one, it waits for each creation and deletion to be complete before running success raters and taking another order. Thus, the order of actions is important, but not their absolute time and temporal density. This might skip some side-effects of a realistic execution. * Similarly, to the above, fake-virt provides simple classes that will not reproduce some behaviours of real-world hypervisors. An explicit Scheduler avoids hosts that had recently failed to allocate a VM, but most likely fake-virt will not mock such behaviour. * Fake-virt should reproduce a real diverse infrastructure instead of x copies of the same flavour. This might be the only, but very important change to the OpenStack codebase. If successful, it could benefit other projects and tests as well. Even though the list of missing features is seemingly larger, the most important parts of the program are already there, so we hope to finish the MVP development in a relatively short amount of time. We are going to publish it as FOSS in either case, but as mentioned your observations would be very much welcome at this stage. I am also open to answering more questions about the project. Kind regards Dominik Danelski

3 3