October 2022 - openstack-discuss - lists.openstack.org

[neutron] Switching the ML2 driver in-place from linuxbridge to OVN for an existing Cloud
by Christian Rohmann 26 Nov '22

26 Nov '22

Hello openstack-discuss and neutron experts, I'd like to ask for your input and discussion on the idea of changing the ML2 driver for an existing cloud, read: changing a tire while still riding the bike. I actually like to find out if it's feasible to switch from the trusted Linuxbridge driver to the more modern SDN stack of OVN - in place. With all existing user networks, subnets, security groups and (running) instances in the database already. And while I know there a push of migrating from OVS to OVN and a clear migration path documented (https://docs.openstack.org/neutron/latest/ovn/migration.html), but they are much more similar in their data plane. And to get this out of the way: I am not asking to do be able to do this without downtime, interruptions, migrations or any active orchestration of the process. I just want to know all the possible options apart from setting up a new cloud and asking folks to migrate all of their things over... 1) Are the data models of the user managed resources abstract (enough) from the ML2 used? So would the composition of a router, a network, some subnets, a few security group and a few instances in a project just result in a different instantiation of packet handling components, but be otherwise transparent to the user? 2) What could be possible migration strategies? While it might be a little early to think about the actual migration steps. But just to consider more than a full cloud shutdown following a cold start with modified neutron config then using the OVN ML2. I know there is more than just the network and getting a virtual layer 2 network. There are DHCP, DNS and the metadata service and last but not least the gateway / router and the security groups. But if using VXLAN for OVN (as Linuxbridge uses currently) could the shift from one implementation to the other potentially be done node by node? Or project by project by changing the network agents over to nodes already running OVN? If not, is there any other hot cut-over approach that would avoid having to shutdown all the instances, but only cause them some network downtime (until the next DHCP renew or similar?) Has anybody ever done something similar or heard about this being done anywhere? Thanks for your time and input, Christian

3 5

[all][stable][ptl] Propose to EOL Queens series
by Elõd Illés 25 Nov '22

25 Nov '22

Hi, As more and more teams decide about moving their Queens branches to End of Life, it looks like the time has come to transition the complete Queens stable release for every project. The reasons behind this are the following things: - gates are mostly broken - minimal number of bugfix backports are pushed to these branches - gate job definitions are still using the old, legacy zuul syntax - gate jobs are based on Ubuntu Xenial, which is also beyond its public maintenance window date and hard to maintain - lack of reviews / reviewers on this branch Based on the above, if no objection comes from teams, then I'll start the process of EOL'ing Queens stable series. Please let the community know what you think, or indicate if any of the projects' stable/queens branch should be kept open in Extended Maintenance. Thanks, Előd Illés irc: elodilles @ #openstack-stable / #openstack-release

1 1

[designate] How to avoid NXDOMAIN or stale data during cold start of a (new) machine
by Christian Rohmann 23 Nov '22

23 Nov '22

Hello openstack-discuss, I have a designate setup using bind9 as the user-serving DNS server. When starting a machine with either very old or no zones at all, NXDOMAIN or other actually stale data is sent out to clients as designate is not done doing an initial full sync / reconciliation. * What is the "proper" way to tackle this cold-start issue and to keep the bind from serving wrong data? ** Did I miss on any options to handle this startup case? * What is the usual runtime for an initial sync that you observe in case the backend DNS server has no zones at all anymore? Regards Christian

2 7

[all][openstack-dev][ptls] Migrating devstack jobs to Jammy (Ubuntu LTS 22.04)
by Dmitriy Rabotyagov 21 Nov '22

21 Nov '22

Hi everyone, According to a 2023.1 community-wide goal [1], base-jobs including but not limited to devstack-minimal, devstack, devstack-ipv6, devstack-multinode, tox, will be switched from Ubuntu Focal (20.04) to Jammy (22.04). That will also bring python 3.10 as a default python interpreter for these jobs. Migration will affect most of the projects, so to make it smooth, we will split the process into the following steps: 1. Patches for switching base jobs to Jammy will be proposed and marked as WIP (Work In Progress). It will allow projects to check if their code is compatible without breaking gates. These patches are ready and you can check them out here [2] 2. All projects should ensure that the code is compatible and merge changes if required. To properly track and test these changes you should: * Create a DNM patch that will contain in its commit message: "Depends-On: https://review.opendev.org/c/openstack/tempest/+/861110" to consume new nodesets. You can check the sample patch for Nova below [3] * In case of any job failure due to Ubuntu 22.04 or Python 3.10, land requires changes to support this new distribution * If you have overridden nodeset for your project jobs, ensure that you also have jobs against Ubuntu 22.04 and these jobs set to voting. * Set topic to all related changes to "migrate-to-jammy" * Please use etherpad [4] for tracking patches, bug reports, current status, or any other activity related to this topic 3. On R-18, which is the first 2023.1 milestone that will happen on the 18th of November 2022, base-jobs patches mentioned in step 1 will be merged. Please ensure you have verified compatibility for your projects and landed the required changes if any were needed before this date otherwise, they might fail. Please, do not hesitate to raise any questions or concerns. [1] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubunt… [2] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/861116 https://review.opendev.org/c/openstack/tempest/+/861110 https://review.opendev.org/c/openstack/devstack/+/860795 [3] https://review.opendev.org/c/openstack/nova/+/861111 [4] https://etherpad.opendev.org/p/migrate-to-jammy

2 2

[kolla-ansible][Yoga] Deployment stuck
by wodel youchi 16 Nov '22

16 Nov '22

Hi, I am trying to deploy a new platform using kolla-ansible Yoga and I am trying to upgrade another platform from Xena to yoga. On both platforms the prechecks went well, but when I start the process of deployment for the first and upgrade for the second, the process gets stuck. I tried to tail -f /var/log/kolla/*/*.log but I can't get hold of the cause. In the first platform, some services get deployed, and at some point the script gets stuck, several times in the modprobe phase. In the second platform, the upgrade gets stuck on : Escalation succeeded [204/1859] <20.3.0.28> (0, b'\n{"path": "/etc/kolla/cron", "changed": false, "diff": {"before": {"path": "/etc/kolla/cro n"}, "after": {"path": "/etc/kolla/cron"}}, "uid": 0, "gid": 0, "owner": "root", "group": "root", "mode": "07 70", "state": "directory", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 70, "invocation": {"module_ args": {"path": "/etc/kolla/cron", "owner": "root", "group": "root", "mode": "0770", "recurse": false, "force ": false, "follow": true, "modification_time_format": "%Y%m%d%H%M.%S", "access_time_format": "%Y%m%d%H%M.%S", "unsafe_writes": false, "state": "directory", "_original_basename": null, "_diff_peek": null, "src": null, " modification_time": null, "access_time": null, "seuser": null, "serole": null, "selevel": null, "setype": nul l, "attributes": null}}}\n', b'') ok: [20.3.0.28] => (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True , 'image': '20.3.0.34:4000/openstack.kolla/centos-source-cron:yoga', 'environment': {'DUMMY_ENVIRONMENT': 'ko lla_useless_env', 'KOLLA_LOGROTATE_SCHEDULE': 'daily'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_f iles/:ro', '/etc/localtime:/etc/localtime:ro', '', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) => { "ansible_loop_var": "item", "changed": false, "diff": { "after": { "path": "/etc/kolla/cron" }, "before": { "path": "/etc/kolla/cron" } }, "gid": 0, "group": "root", How to start debugging the situation. Regards.

1 5

Zun connector for persistent shared files system Manila
by Vaibhav 13 Nov '22

13 Nov '22

Hi, I am using zun for running containers and managing them. I deployed cinder also persistent storage. and it is working fine. I want to mount my Manila shares to be mounted on containers managed by Zun. I can see a Fuxi project and driver for this but it is discontinued now. With Cinder only one container can use the storage volume at a time. If I want to have a shared file system to be mounted on multiple containers simultaneously, it is not possible with cinder. Is there any alternative to Fuxi. is there any other mechanism to use docker Volume support for NFS as shown in the link below? https://docs.docker.com/storage/volumes/ Please advise and give a suggestion. Regards, Vaibhav

2 4

[neutron][release] Proposing to EOL Queens, Rocky and Stein (all Neutron related projects)
by Rodolfo Alonso Hernandez 11 Nov '22

11 Nov '22

Hello: In the last PTG, the Neutron team has decided [1] to move the stable branches Queens, Rocky and Stein to EOL (end-of-life) status. According to the steps to achieve this [2], we need first to announce it. That will affect all Neutron related projects. The patch to mark these branches as EOL will be pushed in one week. If you have any inconvenience, please let me know in this mail chain or in IRC (ralonsoh, #openstack-neutron channel). You can also contact any Neutron core reviewer in the IRC channel. Regards. [1]https://etherpad.opendev.org/p/neutron-antelope-ptg#L131 [2] https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-l…

1 1

[keystone][cache] How to tune role cache
by Eugen Block 08 Nov '22

08 Nov '22

Hi *, one of our customers has two almost identical clouds (Victoria), the only difference is that one of them has three control nodes (HA via pacemaker) and the other one only one control node. They use terraform to deploy lots of different k8s clusters and other stuff. In the HA cloud they noticed keystone errors when they purged a project (cleanly) and started the redeployment immediately after that. We did some tests to find out which exact keystone cache it is and it seems to be the role cache (default 600 seconds) which leads to an error in terraform, it reports that the project was not found and refers to the previous ID of the project. The same deployment seems to work in the single-control environment without these errors, it just works although the cache is enabled as well. I already tried to reduce the cache_time to 30 seconds but that doesn't help (although it takes more than 30 seconds until terraform is ready after the prechecks). But the downside of disabling the role cache entirely leads to significantly longer response times when using the dashboard or querying the APIs. Is there any way to tune the role cache in a way so we could have both a reasonable performance as well as being able to redeploy projects without a "sleep 600"? Any comments or recommendations are appreciated! Regards, Eugen

1 1

(openstack-ansible) Container installation in openstack
by Adivya Singh 07 Nov '22

07 Nov '22

Hi Team, Any input on this, to install container service in openstack using ansible. standard global parametre Regards Adivya Singh

2 5

[Kolla-ansible][Xena] How to update an Openstack deployment with new containers?
by wodel youchi 01 Nov '22

01 Nov '22

Hi, The documentation of kolla-ansible Xena does not talk about updating an existing Xena deployment with new containers. Could you please help with this? I found some lines about that in Yoga version, saying that, to update an existing deployment you have to : 1 - Update kolla-ansible it self : $ source xenavenv (xenavenv) $ pip install --upgrade git+ https://opendev.org/openstack/kolla-ansible@stable/xena 2 - Update the container Images with Docker pull 3 - Update my local registry if I am using one, and in my case I am, so I deleted the registry images then I recreated them. 4 - Then finally deploy again (xenavenv) $ kolla-ansible -i multinode deploy Is this the right procedure? because I followed the same procedure and I think it didn't change anything. For example I am taking nova-libvirt container as and example, in my local registry I have this : [root@rcdndeployer2 ~]# docker images | grep nova-libvirt 192.168.2.34:4000/openstack.kolla/centos-source-nova-libvirt xena 5be83d680102 31 hours ago 2. 34GB quay.io/openstack.kolla/centos-source-nova-libvirt xena *5be83d680102* *31 hours ago * 2. 34GB [root@rcdndeployer2 ~]# docker inspect -f '{{ .Created }}' *5be83d680102 * *2022-10-25*T02:33:13.172550584Z But in my compute nodes I have this : root@computehci24 ~]# docker ps | grep nova-lib b56a12bfd482 192.168.2.34:4000/openstack.kolla/centos-source-nova-libvirt:xena "dumb-init --single-…" *5 months ag**o Up Up 5 months (healthy) nova_libvirt* Regards.

1 1