December 2023 - openstack-discuss

[openstack-ansible] December and January meetings schedule
by Neil Hanlon 13 Dec '23

13 Dec '23

Hello all, and happy holidays :) The OSA team will be skipping our regularly scheduled meetings on December 26th and January 2nd due to the Christmas/New Year festivities. We'll be meeting once more before the end of the year, next Tuesday, December 19th. Thank you, and I wish you all the best in 2024! Cheers, Neil

1 0

System configuration in OpenStack
by lostghost1＠yandex.ru 12 Dec '23

12 Dec '23

Hi all, Sorry if this has been asked already, but is it feasible to use OpenStack to configure running systems - including virtual instances, containers, and the physical nodes themselves. I know that conceptually OpenStack takes a hands-off approach - it deals with images in abstract and further configuration is delegated to tools such as Ansible. Same for bare metal nodes - you provision them with Ironic, flash an image to them, and that's where configuaration ends for OpenStack. At least … [View More]

3 6

[publiccloud-sig] Reminder - next meeting December 13th - 0800 UTC
by Felix Kronlage-Dammers 12 Dec '23

12 Dec '23

Hi everyone, This Wednesday the next meeting of the public cloud sig is going to happen. Since we do every other meeting as a videosession, we’ll be meeting at 0800 UTC here: https://conf.scs.koeln:8443/OIF-public-cloud-sig We’re going to continue our discussion on user stories for the discoverability of flavors and alike. Please see https://etherpad.opendev.org/p/publiccloud-sig-meeting for further infos. See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG … [View More]

1 0

Issuse with OVN DVR generate too much flow on compute node
by Sang Tran Quoc 12 Dec '23

12 Dec '23

Hi OpenStack Community, I've indicated very weird behavior about ovn flows when VM is associated with vrouter, I use one new compute node with only 1 VM for testing and result as below: * When VM attach to vRouter without external network: Only add new 2047 flows. * When vRouter attach to external network (provider-net2): It will generate 147k flows immediately ? * We've tested with VM direct attach to provider-net2 also generate 147k flows. [cid:image001.png@01DA2D02.780FB210] … [View More]

1 0

Re: [URGENT] - Help needed with open stack services hung
by Karl Kloppenborg 12 Dec '23

12 Dec '23

Hey team, Thanks for those who replied off list. Thankfully we figured it out, it was due to underlying resolver config mysterious mulching AAAA records instead of requesting A records… Probably goes to say, Openstack needs to have some seriously heavier logging 😃 haha. Thanks! Karl. From: Dmitriy Rabotyagov <noonedeadpunk(a)gmail.com> Date: Tuesday, 12 December 2023 at 5:35 pm To: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au> Subject: Re: [URGENT] - Help needed with open … [View More]stack services hung Hey, > stopping services from processing requests I assume this means that services are started and do not report any errors in logs? What happens with requests themselves? Are they answered with error or just hang? Also how environment was deployed? Some deployment tooling or manually with packages, k8s? On Tue, Dec 12, 2023, 05:30 Karl Kloppenborg <kkloppenborg(a)resetdata.com.au<mailto:kkloppenborg@resetdata.com.au>> wrote: Hi Openstack Community, Unfortunately we’ve had a weird situation arise where one of our regions has just stopped dead in it’s tracks, all management systems have been rebooted but there seems to be something stopping services from processing requests. Database, RabbitMQ Memcached have all been individually tested and I cannot find fault with them. Could any openstack consultants who are available for immediate emergency work please contact me off list. Thanks, Karl. Kind Regards, Karl Kloppenborg BCompSc, CNCF-[KCNA, CKA, CKAD], LFCE, CompTIA Linux+ XK0-004 Chief Technology Officer m: +61 437 239 565<mailto:+61%20437%20239%20565> | e: kkloppenborg(a)resetdata.com.au<mailto:kkloppenborg@resetdata.com.au> | w: https://resetdata.com [signature_2024530915][Image] This email transmission is intended only for the addressee / person responsible for delivery of the message to such person and may contain confidential or privileged information. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you, nor may you use, review, disclose, disseminate or copy any information contained in or attached to it. Whilst this email has been checked for viruses, the sender does not warrant that any attachments are free from viruses or other defects. You assume all liability for any loss, damage or other consequences which may arise from opening or using the attachments. If you received this e-mail in error please delete it and any attachments and kindly notify us by immediately sending an email to contact(a)resetdata.com.au<mailto:contact@resetdata.com.au> [View Less]

1 0

[tripleo][ironic][wallaby] Partition Table: loop
by Alexey Kashavkin 12 Dec '23

12 Dec '23

I have an issue with nodes provisioning. All my nodes are getting a partition table - loop. I forcefully specified 'disk_label:gpt/msdos' in the capabilities properties, but it didn't help. I can't figure out at what moment it creates a partition for the entire disk size: Model: Virtio Block Device (virtblk) Disk /dev/vda: 53.7GB Sector size (logical/physical): 512B/512B Partition Table: loop Disk Flags: Number Start End Size File system Flags 1 0.00B 53.7GB 53.7GB xfs … [View More]

1 0

[URGENT] - Help needed with open stack services hung
by Karl Kloppenborg 12 Dec '23

12 Dec '23

Hi Openstack Community, Unfortunately we’ve had a weird situation arise where one of our regions has just stopped dead in it’s tracks, all management systems have been rebooted but there seems to be something stopping services from processing requests. Database, RabbitMQ Memcached have all been individually tested and I cannot find fault with them. Could any openstack consultants who are available for immediate emergency work please contact me off list. Thanks, Karl. Kind Regards, Karl … [View More]

1 0

Individual Board Director nominations: 3d left
by Kurt Garloff 12 Dec '23

12 Dec '23

Hi, the Open Infra communities have many strengths. One of them is a well-balanced board of directors that has representatives from all member classes, the Platinum members, the Gold members, and from individuals [1] that are part of the communities and elected -- each class contributing 1/3 of the seats.[4] I am worried about the individual class though: There are currently only 3 candidates that have enough nominations to be on the ballot [2]. 3 more needing a few more nominations. … [View More]

2 1

[kolla] [2023.1] [masakari] Orphaned instances on node shutdown
by Rob Jefferson 12 Dec '23

12 Dec '23

Folks, I'm running into an issue with instances appearing to be running, even though they shouldn't be. The environment in this case is a virtualized OpenStack Kolla (2023.1) cluster, with three compute nodes and three control plane nodes. I am trying to see what happens when I create and terminate instances, in an attempt to figure out failover behavior. The three computes are in a single segment. I created four instances: +--------------------+--------+---------------------------+ | Name … [View More] | Status | Host | +--------------------+--------+---------------------------+ | rj_testinstance_04 | ACTIVE | devtest-libvirt1-muon1005 | | rj_testinstance_03 | ACTIVE | devtest-libvirt1-muon1004 | | rj_testinstance_02 | ACTIVE | devtest-libvirt1-muon1006 | | rj_testinstance | ACTIVE | devtest-libvirt1-muon1005 | +--------------------+--------+---------------------------+ I decided to see what would happen if I went in and shutdown devtest-libvirt1-muon-1005 (which has two instances running). So I ssh'd into the machine and entered (as root): `echo c > /proc/sysrq-trigger` to force a crash, which duly occurred. I then made sure the host was seen as down: +---------------------------+-------+ | Hypervisor Hostname | State | +---------------------------+-------+ | devtest-libvirt1-muon1005 | down | | devtest-libvirt1-muon1004 | up | | devtest-libvirt1-muon1006 | up | +---------------------------+-------+ But even after this, the instances seem to still be up, including on the now crashed machine: +--------------------+--------+---------------------------+ | Name | Status | Host | +--------------------+--------+---------------------------+ | rj_testinstance_04 | ACTIVE | devtest-libvirt1-muon1005 | | rj_testinstance_03 | ACTIVE | devtest-libvirt1-muon1004 | | rj_testinstance_02 | ACTIVE | devtest-libvirt1-muon1006 | | rj_testinstance | ACTIVE | devtest-libvirt1-muon1005 | +--------------------+--------+---------------------------+ I would have expected the instances to have been regenerated on other compute hosts, or some sort of error. The nodes are on the same segment, so Masakari should have at least attempted to roll over the instances to other computes in the segment. But checking the instance details suggests that nothing has happened. The instances are just... lost. The Masakari host monitor log file reflects reality: 2023-12-11 19:21:49.953 8 INFO masakarimonitors.hostmonitor.host_handler.handle_host [-] 'devtest-libvirt1-muon1005' is 'offline' (current: 'offline'). Needless to say, I'm very confused here and am sure what's going on. Any advice would help. Thanks, Rob [View Less]

1 1

[neutron][ovn] Outbound SNAT behavior
by James Denton 12 Dec '23

12 Dec '23

Hi everyone, While looking at an OVS->OVN migration, we came across a use-case that doesn’t provide 1:1 compatibility and I wanted to check with the community to see if there were any suggestions or ideas on how to proceed. In an OVS-based environment, we have users with a typical layout: <external provider network> | <neutron router> | | <tenantnet1> <tenantnet2> The users have implemented static … [View More]routes on the Neutron router using the extraroutes API, which works great, and the default SNAT behavior implemented within the qrouter namespace handles any network traffic egressing the router, not just directly-connected tenant networks (see iptables entries below): -A neutron-l3-agent-snat -j neutron-l3-agent-float-snat -A neutron-l3-agent-snat -o qg-700f84b0-6b -j SNAT --to-source 168.47.160.223 --random-fully -A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 168.47.160.223 --random-fully -A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat OVN, on the other hand, appears to implement snat rules in the logical router based on directly-connected tenant networks, which means that traffic sourcing from other networks (not managed by Neutron but routable within a Neutron router, including to a VM or appliance) is not source NAT’d and egresses the router as its real IP. This behavior results in the traffic being effectively blackholed, as there’s no return route and no real desire to have exposed those IPs anyway. Here’s an example of the nat list which includes the directly connected tenant network: root@lab-controller01:~# ovn-nbctl lr-nat-list neutron-edfd7115-5a81-46b7-9692-fe227d395b0b TYPE GATEWAY_PORT EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT snat 192.168.3.33 10.254.254.0/24 The extraroute in the router is: destination: 172.16.0.0/24, nexthop: 10.254.254.6 I can add an entry manually, which allows the snat for the secondary network to work as expected, but results in some other issues where the snat(s) don’t get added back on a disable/enable of the router: root@lab-controller01:~# ovn-nbctl lr-nat-add neutron-edfd7115-5a81-46b7-9692-fe227d395b0b snat 192.168.3.33 172.16.0.0/24 root@lab-controller01:~# ovn-nbctl lr-nat-list neutron-edfd7115-5a81-46b7-9692-fe227d395b0b TYPE GATEWAY_PORT EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT snat 192.168.3.33 10.254.254.0/29 snat 192.168.3.33 172.16.0.0/24 I understand this is probably not a supported usecase, but one that worked nonetheless, so any suggestions on how to mirror this functionality in a supported way would be greatly appreciated. Thanks, James [View Less]

1 0