March 2020 - openstack-discuss - lists.openstack.org

[nova] API updates week 20-13
by Ghanshyam Mann 27 Mar '20

27 Mar '20

Hello Everyone, Please find the Nova API updates of this week. Please add if I missed any BPs/API related work. API Related BP : ============ COMPLETED: 1. Add image-precache-support spec: - https://blueprints.launchpad.net/nova/+spec/image-precache-support Code Ready for Review: ------------------------------ 1. Nova API policy improvement - Topic: https://review.opendev.org/#/q/topic:bp/policy-defaults-refresh+(status:ope… - Weekly Progress: Lot of Policies work merged and I am working on remaining ones. - Review guide over ML - http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008504.h… 2. Non-Admin user can filter their instance by availability zone: - Topic: https://review.opendev.org/#/q/topic:bp/non-admin-filter-instance-by-az+(st… - Weekly Progress: Nova change is merged, novaclient change is in-progress - https://review.opendev.org/#/c/713089/ 3. Boot from volume instance rescue - Topic: https://review.opendev.org/#/q/topic:bp/virt-bfv-instance-rescue+(status:op… - Weekly Progress: Code Review is in progress. 4. Add action event fault details -Topic: https://review.opendev.org/#/q/topic:bp/action-event-fault-details+(status:… - Weekly Progress: Few discussion on how we can minimize the info leak to non-admin. discussion is going on. Almost good. 5. Support re-configure deleted_on_termination in server -Topic: https://review.opendev.org/#/q/topic:bp/destroy-instance-with-datavolume+(s… - Weekly Progress: Discussion happened not to use PATCH for this and use existing PUT method. gibi will discuss this in today nova meeting on process and FF wise. Specs are merged and code in-progress: ------------------------------ ------------------ - None Spec Ready for Review or Action from Author: --------------------------------------------------------- -Marking as None here as Ussuri spec deadline is already over. Others: 1. None Bugs: ==== I started fixing policy bugs while working on policy-defaults-refresh BP. 5 bugs have been identified till now and fix up for review. - https://bugs.launchpad.net/nova/+bugs?field.tag=policy-defaults-refresh NOTE- There might be some bug which is not tagged as 'api' or 'api-ref', those are not in the above list. Tag such bugs so that we can keep our eyes. - https://etherpad.openstack.org/p/nova-bug-triage-ussuri -gmann

1 0

Regarding Split network plane for live migration
by Md. Hejbul Tawhid MUNNA 27 Mar '20

27 Mar '20

Hi, We are using Openstack Rocky . We are observing internal network is using while we do live migration. our internal network interface is 1G. So interface getting full and migration never getting complete. (nova server-migration-list ) 'Remaining Memory Bytes' increasing and decreasing but not getting completed. We have 10G interface for public/external network. I have tried following solution but no luck. still using internal network https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/s… Some Information: [DEFAULT] my_ip = Internal_network_IP [libvirt] live_migration_uri=qemu+ssh://nova@%s.external /system?keyfile=/var/lib/nova/.ssh/id_rsa&no_verify=1 Please let us know if any further information is required. Please advice us if we can solve the issue. Regards, Munna

3 4

[openstack-ansible][elections] Nomination
by Mohammed Naser 27 Mar '20

27 Mar '20

I'd like to announce my candidacy for OpenStack Ansible PTL. As usual, I want to go over all the points I talked about last time and their progress. # Simplifying scenarios for usage and testing We probably haven't progressed much in this, our testing has improved, we did add upgrade testing but unfortuantely that hasn't been as stable. # Using integrated repository for testing and dropping role tests This is pretty much done for the expect of a few corner projects. # Progress on switching all deployments to use Python3 This is also done with some packaging workarounds to make it work with CentOS 7. # Eventual addition of CentOS 8 option We made some progress in this, but it still needs a lot more work, big thanks to jrosser for driving a bunch of this work too. # Reduction in number of config variables (encouraging overrides) This has been done over a few roles now that are using overrides in hopes of minimizing the number of variables to speed up runs. # Increase cooperation with other deployment projects (i.e. TripleO) The Ansible SIG is moving and alive, TripleO uses python_venv_build and os_tempest as well as our connection plugins, so yay. We've worked on getting some new cores as well and making sure the project continues to be sustainable. Thank you for your consideration.

1 0

[tc][elections] Nomination
by Mohammed Naser 27 Mar '20

27 Mar '20

Hi everyone, I want to run again for the OpenStack technical committee for the upcoming cycle. I have been a long time operator of OpenStack with a focus on never using a local for, which has resulted in my involvement with many different projects by making quality-of-life improvements for these services that can help other deployers. I've contributed and worked with different sizes of teams inside OpenStack. >From our big projects such as Nova to projects providing specific services such as Magnum and pretty much most deployments tools at this point. On the operator side, I've helped run one of the longest-running OpenStack public clouds and many private clouds. I've also interacted with many other OpenStack operators that are not as involved in our community, which gives me a significant exposure to our operator story and what that looks like outside our community of known operators. I believe it's time now for me to start driving significant changes in our community to simplify the operator experience, and I think that should start from the technical committee. We're seeing the world evolve around us with new technologies, but we've remained relatively stable and stuck to our "known and trusted" stack. Operators have historically had a lot of frustrations with systems like RabbitMQ and their clustering issues, which end up making the OpenStack experience disappointing. There are many new ways of doing RPC, which we can leverage and help make it even easier to run OpenStack (things such as gRPC). We've also had a history of making it complicated to use containers to deploy OpenStack and historically refused to ship "official" containers. The OpenDev community has built our a step of tooling that can allow us to ship containers very quickly, with nothing more than a well written bindep.txt file and a simple multi-stage build. It's about time that we start having official images and easy ways to deploy OpenStack to reduce those barriers. We define OpenStack as a "cloud operating system." We struggle to deliver much more than the essential infrastructure components. The drive behind OpenStack was to allow people to get more than the necessary infrastructure components. Still, large amounts of different efforts lead us into complicated ways of having to wrangle VMs to run services. With popular container orchestrators and schedulers, OpenStack has a perfect platform to be able to launch those services. By setting up a proper framework, we can start delivering many services such as database as a service, memory store as a service, and so much more by building out Kubernetes operators. It also opens up a whole domain of possible contributors that can use those operators without OpenStack. In closing, I don't expect everyone to agree with all the different ideas I'm putting out. We may only drive just a few of them, or maybe none of them. However, I want to convince you that we need to change. I can understand they sound like big leaps, and they may seem uncomfortable. However, they are genuinely some of the things we need to do to continue to be relevant and put us at the leading edge of technology like we used to be years ago. Thank you for reading this. Regards, Mohammed

1 0

[keystone][horizon] Two approaches to "multi-region" OpenStack deployment
by Krzysztof Klimonda 26 Mar '20

26 Mar '20

Hi, I’ve been spending some time recently thinking about best approach to some sort of multi-region deployment of OpenStack clouds. The two approaches that I’m currently evaluating are shared keystone database (with galera cluster spanning three locations) and shared-nothing approach, where external component is responsible for managing users, projects etc. Shared keystone database seems fairly straightforward from OS point of view (I’m ignoring galera replication over WAN woes for the purpose of this discussion) until I hit 3 regions. Additional regions must either reuse “global” keystone adding latency everywhere, or we need a way to replicate data from “master” galera cluster to “slave” clusters, and route all database write statements back to the master galera cluster, while reading from local asynchronous replica. This has me worried somewhat, as doing that that into eventually-consistent deployment of sort. Services deployed in regions with asynchronous replication can no longer depend on the fact that once transaction is finished, consecutive reads will return up-to-date state. I can imagine scenarios where, as an example, trust is setup for heat, but that fact is not replicated back to the database by the time heat tries to issue a token based on that trust and the process fails. The other approach would be to keep keystone databases completely separate, and have something external to openstack manage all those resources. While not sharing keystone database between regions sidesteps the issue of scalability, and the entire setup seems to be more resilient to failures, it’s not without its own drawbacks: * In this setup Horizon can no longer switch between regions without additional work (see notes below) * There is no longer single API entrypoint to the cloud * Some Keystone API operations would have to be removed from users via custom policy - for example, managing user assignment to projects (for users who have domain admin role) Additional thoughts Could horizon be modified to switch endpoints based on the region selected in the UI? Is the token reissued when region is changed in horizon, or is single token used? I’m assuming it’s the former given my understanding that when projects are changed, a new token is issued - but perhaps the initial token is always used to issue project-scoped tokens for Horizon? In the second scenario, with separate keystone databases, a backend for keystone could be created that proxies some operations (like aforementioned user assignment) back to the external manager so that it can be propagated to other clouds. Does that even make sense? In the end I’m reaching out in hope that someone could chime in based on their experience - perhaps I’m missing a better approach, or making wrong assumptions in my email, especially around asynchronous replication of keystone database and its effect on services in regions that may not have up-to-data view of the databas. Or perhaps trying ot synchronize keystone state by external tool is not really worth the additional effort that would require. -Chris

6 8

[tripleo] ptg topics
by Wesley Hayutin 26 Mar '20

26 Mar '20

Greetings, It's that time of year to start collecting topics for the PTG [1]. The in person PTG has been switched to a virtual meeting [2]. So the bad news is that the world is upside down atm, but the good news is that you may have a better lunch at the sessions ( I kid.. ) You can help plan the virtual ptg by signing up @ https://etherpad.openstack.org/p/Virtual_PTG_Planning Thanks all!! [1] https://etherpad.openstack.org/p/tripleo-victoria-topics [2] http://lists.openstack.org/pipermail/foundation/2020-March/002854.html

1 0

[all] Collecting Virtual Midcycle Best Practices
by Kendall Nelson 26 Mar '20

26 Mar '20

Hello Everyone! I wanted to collect best practices and pitfalls to avoid wrt projects experiences with virtual midcycles. I know of a few projects that have done them in the past and with how travel is hard for a lot of people right now, I expect more projects to have midcycles. I think it would be helpful to have all of the data we can collect in one place for those not just new to virtual midcycles but the whole community. I threw some categories into this etherpad[1] and filled in some options. Please add to it :) -Kendall (diablo_rojo) [1] https://etherpad.openstack.org/p/virtual-midcycle-best-practices

4 8

[tripleo] rdo registry migration
by Wesley Hayutin 26 Mar '20

26 Mar '20

Greetings, FYI.. the rdo registry is migrating from RDO to Vexxhost. You may find some of your jobs hitting "retry_limit" [1] This was caused by a default to log into registries for jobs running in rdo. The default has been updated. Jobs should return to their "GREENISH" status shortly [1] https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-…

1 0

[heat][magnum][tripleo] Official build and test plan for heat container agent
by Rico Lin 26 Mar '20

26 Mar '20

Dear all tl;dr: Shall we build, publish, and test heat container agents on heat-agents directly?:) Please help to review https://review.opendev.org/#/q/topic:build-container-agent+(status:open) Current I think we encounter with following issues: ** No functional/scenario test for heat-agents but only unit tests* Right now, whatever patch people send up in heat-agents, we only vote against unit tests. This makes no sense for long term maintenance. Which makes me thinking of the second issue. ** No enable scenario tests for software deployment:* We have tests defined, but since we didn't build any customized image. We didn't enable it for a long time. To build a container image for heat-agents (aka heat container agent) is (IMO) the best way for us to enable these teats. Which we can also use it to cross gating on heat and heat-agents (or even heat-template). The no much to change for the test itself but only update the config script to enable the container. ** Can't guarantee cross-project usage:* We currently provide no guarantee for any usage of heat agents from other projects, especially Magnum and TripleO (I didn't personally check if TripleO still using it, but since they still have experiment CI running) who uses heat container agents. It will be nice if we provide basic images for other teams can build their additional requirements. To resolve the above issues, I propose [1]. Which a lot of logic was copied from the current magnum heat container agent scripts [2] so we can start with a higher possibility to allow magnum to reuse our works. For a more detail approach in [1]. The patch defines a build job in `check` pipeline. And a publish job in `post-check`. Uses zuul secrets to save docker authN information. As for the account, I created a new `openstackheat` account [3] for it (I specifically recall there ware discussion about having an official image hub for OpenStack, but not quite sure what happened since). The current approach also can support multiple architectures (like amd64 arm arm64 ppc64le s390x, etc). But for now, I think we should have tested before we publish for other architectures. You can run the build or publish job easily by running `make build-images-all-archs` to build images for all architectures or `make upload-images-all-archs` to build and publish images for all architectures. [1] https://review.opendev.org/#/q/topic:build-container-agent+(status:open) [2] https://opendev.org/openstack/magnum/src/branch/master/dockerfiles/heat-con… [3] https://hub.docker.com/r/openstackheat -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

1 1

[cinder][qa] propose Liang Fang for devstack-plugin-open-cas-core
by Brian Rosmaita 25 Mar '20

25 Mar '20

Liang Fang is driving the effort to create the devstack-plugin-open-cas and use it to test the cinder/nova volume-local-cache feature with tempest. Since he's our SME on open-cas, I propose that he be made a member of devstack-plugin-open-cas-core. (Currently, the only members of that group are the members of cinder-core and devstack-core.) In the absence of objections, I'll make the change before the next Cinder team meeting (Wednesday, 25 March at 1400 UTC in its new location of #openstack-meeting-alt). cheers, brian

4 4