June 2019 - openstack-discuss - lists.openstack.org

Re: [openstack-dev] Are we ready to put stable/ocata into extended maintenance mode?
by Alex Schultz 03 Mar '21

03 Mar '21

On Tue, Sep 18, 2018 at 1:30 PM Alex Schultz <aschultz(a)redhat.com> wrote: > > On Tue, Sep 18, 2018 at 1:27 PM, Matt Riedemann <mriedemos(a)gmail.com> wrote: > > The release page says Ocata is planned to go into extended maintenance mode > > on Aug 27 [1]. There really isn't much to this except it means we don't do > > releases for Ocata anymore [2]. There is a caveat that project teams that do > > not wish to maintain stable/ocata after this point can immediately end of > > life the branch for their project [3]. We can still run CI using tags, e.g. > > if keystone goes ocata-eol, devstack on stable/ocata can still continue to > > install from stable/ocata for nova and the ocata-eol tag for keystone. > > Having said that, if there is no undue burden on the project team keeping > > the lights on for stable/ocata, I would recommend not tagging the > > stable/ocata branch end of life at this point. > > > > So, questions that need answering are: > > > > 1. Should we cut a final release for projects with stable/ocata branches > > before going into extended maintenance mode? I tend to think "yes" to flush > > the queue of backports. In fact, [3] doesn't mention it, but the resolution > > said we'd tag the branch [4] to indicate it has entered the EM phase. > > > > 2. Are there any projects that would want to skip EM and go directly to EOL > > (yes this feels like a Monopoly question)? > > > > I believe TripleO would like to EOL instead of EM for Ocata as > indicated by the thead > http://lists.openstack.org/pipermail/openstack-dev/2018-September/134671.ht… > Bringing this backup to see what we need to do to get the stable/ocata branches ended for the TripleO projects. I'm bringing this up because we have https://review.openstack.org/#/c/647009/ which is for the upcoming rename but CI is broken and we have no interest in continue to keep the stable/ocata branches alive (or fix ci for them). Thanks, -Alex > Thanks, > -Alex > > > [1] https://releases.openstack.org/ > > [2] > > https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… > > [3] > > https://docs.openstack.org/project-team-guide/stable-branches.html#extended… > > [4] > > https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.… > > > > -- > > > > Thanks, > > > > Matt > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

5 11

[openstack-ansible] dropping suse support
by Mohammed Naser 05 May '20

05 May '20

Hi everyone, I've been trying to avoid writing this email for the longest time ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible. Unfortunately, with the recent layoffs, the OSA team has taken a huge hit in terms of contributors which means that there is far less contributors within the project. This means that we have less resources to go around and it forces us to focus on a more functional, reliable and well tested set of scenarios. Over the past few cycles, there has been effort to add SUSE support to OpenStack Ansible, during the time that we had maintainers, it was great and SUSE issues were being fixed promptly. In addition, due to the larger team at the time, we found ourselves having some extra time where we can help unbreak other gates. Jesse used to call this a "labour of love", which I admired at the time and hoped we continue to do as much as we can of. However, the lack of a committed maintainer for OpenSUSE has resulted in constantly failing jobs[1][2] (which were moved to non-voting, wasting CI resources as no one fixed them). In addition, it's causing several gate blocks for a few times with no one really finding the time to clean them up. We are resource constrained at this point and we need the resource to go towards making the small subset of supported features functional (i.e. CentOS/Ubuntu). We struggle with that enough, and there seems to be no deployers that are running SUSE in real life at the moment based on bugs submitted. With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore. ..Really wish I didn't have to write this email Mohammed [1]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct… [2]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct…

6 9

[kolla-ansible] migration
by Ignazio Cassano 02 Oct '19

02 Oct '19

Hello, Anyone have tried to migrate an existing openstack installation to kolla containers? Thanks Ignazio

5 16

[nova] critical bug around reload/upgrades
by Mohammed Naser 23 Aug '19

23 Aug '19

Hello: I've discussed this for quite sometime with Dan over IRC and a bit with Zane as well, but basically, Nova thinks that when it gets a reload (aka SIGHUP), nothing else has occurred. However, oslo.service actually calls stop(), reload() then start() again, which potentially kills all RPC. This has caused a pretty big issue in our gates and it also means that the whole idea behind 'reload nova-compute while upgrading to refresh info' concept is fundamentally broken. I tried to do some work on this here, however, I wasn't really able to get to the bottom of it. There seems to be a decision that needs to be taken in terms of .. do we change what reload() actually means in oslo_service (it actually is more like a restart, not a reload) or does Nova (and other projects) change their implementation in assuming what reload() does? https://review.openstack.org/#/c/641907/ This seems to have been floating around for a really long time, so I'd be happy to work with someone to find the fix (and we can totally test it inside OpenStack Ansible by reloading instead of restarting). Thanks! Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

4 4

[nova][ops] What should the compute service delete behavior be wrt resource providers with allocations?
by Matt Riedemann 22 Aug '19

22 Aug '19

Before [1] when deleting a compute service in the API we did not check to see if the compute service was hosting any instances and just blindly deleted the service and related compute_node(s) records which orphaned the resource provider(s) for those nodes. With [2] we built on that and would cleanup the (first [3]) compute node resource provider by first deleting any allocations for instances still on that host - which because of the check in [1] should be none - and then deleted the resource provider itself. [2] forgot about ironic where a single compute service can be managing multiple (hundreds or even thousands) of baremetal compute nodes so I wrote [3] to delete *all* resource providers for compute nodes tied to the service - again barring there being any instances running on the service because of the check added in [1]. What we've failed to realize until recently is that there are cases where deleting the resource provider can still fail because there are allocations we haven't cleaned up, namely: 1. Residual allocations for evacuated instances from a source host. 2. Allocations held by a migration record for an unconfirmed (or not yet complete) migration. Because the delete_resource_provider method isn't checking for those, we can get ResourceProviderInUse errors which are then ignored [4]. Since that error is ignored, we continue on to delete the compute service record [5], effectively orphaning the providers (which is what [2] was meant to fix). I have recreated the evacuate scenario in a functional test here [6]. The question is what should we do about the fix? I'm getting lost thinking about this in a vacuum so trying to get some others to help think about it. Clearly with [1] we said you shouldn't be able to delete a compute service that has instances on it because that corrupts our resource tracking system. If we extend that to any allocations held against providers for that compute service, then the fix might be as simple as not ignoring the ResourceProviderInUse error and fail if we can't delete the provider(s). The question I'm struggling with is what does an operator do for the two cases mentioned above, not-yet-complete migrations and evacuated instances? For migrations, that seems pretty simple - wait for the migration to complete and confirm it (reverting a cold migration or resize would put the instance back on the compute service host you're trying to delete). The nastier thing is the allocations tied to an evacuated instance since those don't get cleaned up until the compute service is restarted [7]. If the operator never intends on restarting that compute service and just wants to clear the data, then they have to manually delete the allocations for the resource providers associated with that host before they can delete the compute service, which kind of sucks. What are our options? 1. Don't delete the compute service if we can't cleanup all resource providers - make sure to not orphan any providers. Manual cleanup may be necessary by the operator. 2. Change delete_resource_provider cascade=True logic to remove all allocations for the provider before deleting it, i.e. for not-yet-complete migrations and evacuated instances. For the evacuated instance allocations this is likely OK since restarting the source compute service is going to do that cleanup anyway. Also, if you delete the source compute service during a migration, confirming or reverting the resize later will likely fail since we'd be casting to something that is gone (and we'd orphan those allocations). Maybe we need a functional recreate test for the unconfirmed migration scenario before deciding on this? 3. Other things I'm not thinking of? Should we add a force parameter to the API to allow the operator to forcefully delete (#2 above) if #1 fails? Force parameters are hacky and usually seem to cause more problems than they solve, but it does put the control in the operators hands. If we did remove allocations for an instance when deleting it's compute service host, the operator should be able to get them back by running the "nova-manage placement heal_allocations" CLI - assuming they restart the compute service on that host. This would have to be tested of course. Help me Obi-Wan Kenobi. You're my only hope. [1] https://review.opendev.org/#/q/I0bd63b655ad3d3d39af8d15c781ce0a45efc8e3a [2] https://review.opendev.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80 [3] https://review.opendev.org/#/c/657016/ [4] https://github.com/openstack/nova/blob/cb0cfc90e1e03e82c42187ec60f46fb8fd59… [5] https://github.com/openstack/nova/blob/cb0cfc90e1e03e82c42187ec60f46fb8fd59… [6] https://review.opendev.org/#/c/663737/ [7] https://github.com/openstack/nova/blob/cb0cfc90e1e03e82c42187ec60f46fb8fd59… -- Thanks, Matt

6 16

[storyboard] email notification on stories/tasks of subscribed projects
by Akihiro Motoki 22 Aug '19

22 Aug '19

Hi, Is there a way to get email notification on stories/tasks of subscribed projects in storyboard? In launchpad, I configure launchpad bug notifications of all bug changes in interested projects. After migration to storyboard (for example, openstackclient), I failed to find a similar way. Thanks, Akihiro Motoki (irc: amotoki)

5 8

[tripleo][openstack-ansible] Integrating ansible-role-collect-logs in OSA
by Chandan kumar 12 Aug '19

12 Aug '19

Hello, With os_tempest project, TripleO and Openstack Ansible projects started collaborating together to reuse the tools developed by each other to avoid duplicates and enable more collaboration. During Denver Train 2019 PTG, we decided to unifying the CI logs for both the projects by providing a unified experience to the developers while browsing the CI logs making sure we have a similar logs structure for both the projects, so that one can easily navigate through the logs without scratching the heads and also the logs tree structure should say what logs are present where. In TripleO, we have ansible-role-collect-logs[1.] role for the same and in OSA we have logs_collect.sh[2.] script for the same. But once the logs gets collected at each other projects, It is very hard to navigate and find out where is the respective files. A little about ansible-role-collect-logs roles: * A role for aggregating logs from different nodes * Provide a list for files to collect at one places in playbook [3] * Hyperlink the logs with description about log files * Once logs collection is done, it pushes to a particular log server For example, tempest.html or stestr_results.html is the common file for viewing tempest results. In TripleO, we keep it under logs folder but in OSA logs/openstack/aio1-utility/stestr_results.html. If a new user contributes to another project, he tries to follow the same pattern for find logs as he seen in current project. By Keeping the same structure at all places It would be easier. So moving ahead what we are going to do: * Refactor collect-logs role to pass defaults list of files at one place * Pass the list of different logs files based on deployment tools * Put system/containers related commands at one place * Replace the collect_logs.sh script with playbook in OSA and replace it. Thanks for reading, We are looking for the comments on the above suggestion. Links: [1.] https://opendev.org/openstack/ansible-role-collect-logs/ [2.] https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/l… [3.] https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/… Thanks, Chandan Kumar

4 4

[ironic][ops] Taking ironic nodes out of production
by Arne Wiebalck 09 Aug '19

09 Aug '19

Dear all, One of the discussions at the PTG in Denver raised the need for a mechanism to take ironic nodes out of production (a task for which the currently available 'maintenance' flag does not seem appropriate [1]). The use case there is an unhealthy physical node in state 'active', i.e. associated with an instance. The request is then to enable an admin to mark such a node as 'faulty' or 'in quarantine' with the aim of not returning the node to the pool of available nodes once the hosted instance is deleted. A very similar use case which came up independently is node retirement: it should be possible to mark nodes ('active' or not) as being 'up for retirement' to prepare the eventual removal from ironic. As in the example above, ('active') nodes marked this way should not become eligible for instance scheduling again, but automatic cleaning, for instance, should still be possible. In an effort to cover these use cases by a more general "quarantine/retirement" feature: - are there additional use cases which could profit from such a "take a node out of service" mechanism? - would these use cases put additional constraints on how the feature should look like (e.g.: "should not prevent cleaning") - are there other characteristics such a feature should have (e.g.: "finding these nodes should be supported by the cli") Let me know if you have any thoughts on this. Cheers, Arne [1] https://etherpad.openstack.org/p/DEN-train-ironic-ptg, l. 360

5 7

Re: [telemetry] Team meeting agenda for tomorrow
by Joseph Davis 05 Aug '19

05 Aug '19

On 5/8/19 7:12 AM, openstack-discuss-request(a)lists.openstack.org wrote: > Hello Trinh, > Where does the meeting happen? Will it be via IRC Telemetry channel? Or, in > the Etherpad (https://etherpad.openstack.org/p/telemetry-meeting-agenda)? I > would like to discuss and understand a bit better the context behind > the Telemetry > events deprecation. Unfortunately, I have a conflict at that time and will not be able to attend. I do have a little bit of context on the Events deprecation to share. First, you will note the commit message from the commit [0] when Events were deprecated: " Deprecate event subsystem This subsystem has never been finished and is not maintained. Deprecate it for future removal. " I got the impression from jd at the time that there were a number of features in Telemetry, including Panko, that were not really "finished" and that the engineers who had worked on them had moved on to other things, so the features had become unsupported. In late 2018 there was an effort to clean up things that were not well maintained or didn't fit the direction of Telemetry. See also: https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/ Events is one feature that often gets requested, but the use cases and demand for it are not expressed strongly or well understood by most people. If the Telemetry project has demand to de-deprecate Event handling (including Panko), I'd suggest a review of the requirements for event handling and possibly choosing a champion for maintaining the Panko service. Also note: over in Monasca we have a spec [1] for handling Events ingestion which I hope we will be completing in Train. Contributions and comments welcome. :) joseph [0] https://github.com/openstack/ceilometer/commit/8a0245a5b3e1357d35ad6653be37… [1] https://github.com/openstack/monasca-specs/blob/master/specs/stein/approved… > > On Wed, May 8, 2019 at 12:19 AM Trinh Nguyen<dangtrinhnt(a)gmail.com> wrote: > >> Hi team, >> >> As planned, we will have a team meeting at 02:00 UTC, May 9th on >> #openstack-telemetry to discuss what we gonna do for the next milestone >> (Train-1) and continue what we left off from the last meeting. >> >> I put here [1] the agenda thinking that it should be fine for an hour >> meeting. If you have anything to talk about, please put it there too. >> >> [1]https://etherpad.openstack.org/p/telemetry-meeting-agenda >> >> >> Bests, >> >> -- >> *Trinh Nguyen* >> *www.edlab.xyz<https://www.edlab.xyz>* >> >>

14 25

[Airship-Seaworthy] Deployment of Airship-Seaworthy on Virtual Environment
by Anirudh Gupta 05 Aug '19

05 Aug '19

Hi Team, We want to test Production Ready Airship-Seaworthy in our virtual environment The link followed is https://airship-treasuremap.readthedocs.io/en/latest/seaworthy.html<https://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairship-t…> As per the document we need 6 DELL R720xd bare-metal servers: 3 control, and 3 compute nodes. But we need to deploy our setup on Virtual Environment. Does Airship-Seaworthy support Installation on Virtual Environment? We have 2 Rack Servers with Dual-CPU Intel® Xeon® E5 26xx with 16 cores each and 128 GB RAM. Is it possible that we can create Virtual Machines on them and set up the complete environment. In that case, what possible infrastructure do we require for setting up the complete setup. Looking forward for your response. Regards अनिरुद्ध गुप्ता (वरिष्ठ अभियंता) Hughes Systique Corporation D-23,24 Infocity II, Sector 33, Gurugram, Haryana 122001 DISCLAIMER: This electronic message and all of its contents, contains information which is privileged, confidential or otherwise protected from disclosure. The information contained in this electronic mail transmission is intended for use only by the individual or entity to which it is addressed. If you are not the intended recipient or may have received this electronic mail transmission in error, please notify the sender immediately and delete / destroy all copies of this electronic mail transmission without disclosing, copying, distributing, forwarding, printing or retaining any part of it. Hughes Systique accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus.

3 5

Magnum Kubernetes openstack-cloud-controller-manager unable not resolve master node by DNS
by Pawel Konczalski 31 Jul '19

31 Jul '19

Hi, i try to deploy a Kubernetes cluster with OpenStack Magnum but the openstack-cloud-controller-manager pod fails to resolve the master node hostname. Does magnum require further parameter to configure the DNS names of the master and minions? DNS resolution in the VMs works fine. Currently there is no Designate installed in the OpenStack setup. openstack coe cluster template create kubernetes-cluster-template1 \ --image Fedora-AtomicHost-29-20190429.0.x86_64 \ --external-network public \ --dns-nameserver 8.8.8.8 \ --master-flavor m1.kubernetes \ --flavor m1.kubernetes \ --coe kubernetes \ --volume-driver cinder \ --network-driver flannel \ --docker-volume-size 25 openstack coe cluster create kubernetes-cluster1 \ --cluster-template kubernetes-cluster-template1 \ --master-count 1 \ --node-count 2 \ --keypair mykey # kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE kube-system coredns-78df4bf8ff-mjp2c 0/1 Pending 0 36m <none> <none> <none> kube-system heapster-74f98f6489-tgtzl 0/1 Pending 0 36m <none> <none> <none> kube-system kube-dns-autoscaler-986c49747-wrvz4 0/1 Pending 0 36m <none> <none> <none> kube-system kubernetes-dashboard-54cb7b5997-sk5pj 0/1 Pending 0 36m <none> <none> <none> kube-system openstack-cloud-controller-manager-dgk64 0/1 CrashLoopBackOff 11 36m <none> kubernetes-cluster1-vulg5fz6hg2n-master-0 <none> # kubectl -n kube-system logs openstack-cloud-controller-manager-dgk64 Error from server: Get https://kubernetes-cluster1-vulg5fz6hg2n-master-0:10250/containerLogs/kube-…: dial tcp: lookup kubernetes-cluster1-vulg5fz6hg2n-master-0 on 8.8.8.8:53: no such host BR Pawel

2 3

[forum][sdk][nova] Closing compute API feature gaps in the openstack CLI - recap
by Matt Riedemann 29 Jul '19

29 Jul '19

I wanted to give a quick recap of this Forum session for those that couldn't attend and also find owners. Please reply to this thread if you'd like to sign up for any specific item. The etherpad [1] has the details. To restate the goal: "Identify the major issues and functional gaps (up through Mitaka 2.25) and prioritize which to work on and what the commands should do." We spent the majority of the time talking about existing issues with compute API functionality in openstack CLI, primarily boot-from-volume, live migration and lack of evacuate support (evacuate as in rebuild on a new target host because the source host is dead, not drain a host with live migrations [2]). We then talked through some of the microversion gaps and picked a few to focus on. Based on that, the agreements and priorities are: **High Priority** 1. Make the boot-from-volume experience better by: a) Support type=image for the --block-device-mapping option. b) Add a --boot-from-volume option which will translate to a root --block-device-mapping using the provided --image value (nova will create the root volume under the covers). Owner: TBD (on either) 2. Fix the "openstack server migrate" command We're going to deprecate the --live option and add a new --live-migration option and a --host option. The --host option can be used for requesting a target host for cold migration (omit the --live/--live-migration option for that). Then in a major release we'll drop the --live option and intentionally not add a --force option (since we don't want to support forcing a target host and bypassing the scheduler). Owner: TBD (I would split the 2.56 cold migration --host support from the new --live-migration option review-wise) **Medium Priority** Start modeling migration resources in the openstack CLI, specifically for microversions 2.22-2.24, but note that the GET /os-migrations API is available since 2.1 (so that's probably easiest to add first). The idea is to have a new command resource like: openstack compute migration (list|delete|set) [--server <server_id>] Owner: TBD (again this is a series of changes) **Low Priority** Closing other feature gaps can be done on an as-needed basis as we've been doing today. Sean Mooney is working on adding evacuate support, and there are patches in flight (see [3]) for other microversion-specific features. I would like to figure out how to highlight these to the OSC core team on a more regular basis, but we didn't really talk about that. I've been trying to be a type of liaison for these patches and go over them before the core team tries to review them to make sure they match the API properly and are well documented. Does the OSC core team have any suggestions on how I can better socialize what I think is ready for core team review? [1] https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps [2] http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-co… [3] https://etherpad.openstack.org/p/compute-api-microversion-gap-in-osc -- Thanks, Matt

2 2

[dev][magnum] Magnum event notifications not having relevant content related to clusters
by Emnauel Andrecut 23 Jul '19

23 Jul '19

I'm writing about Magnum event notifications that do not contain any relevant information about the related Cluster (missing cluster id, other attributes..) As I've seen the PyCADF library receives no targetId and so the id (and mostly any other id) is randomly generated. Do you plan to add more information about the cluster on create success/pending events. Or would you accept a patch that does that?

3 2

[all] [ptls] [tc] [nova] [neutron] Volunteers that know TeX for PDF community goal
by Alexandra Settle 19 Jul '19

19 Jul '19

Hi all, The work for the Train community goal - PDF support for project docs - is well underway. [1] Now, we're looking for volunteers to help test the implementation. We'll need someone to help build the docs into PDFs and determine things we can fix through tweaks to our docs, or if they're bugs in Sphinx. AKA: We need a troubleshoot artist. If you can volunteer, please add yourself to the wiki table here [2]. I've added neutron and nova specifically here as we need someone who is familiar with the project and it's dependencies to help us get that setup. Any questions? Reach out. Cheers, Alex [1] https://review.opendev.org/#/q/topic:build-pdf-docs+(status:open+OR+status:… [2] https://wiki.openstack.org/wiki/Documentation#PDF_for_Project_Docs_-_Commun…

8 12

Barbican support for Window
by Garaga Rayudu 18 Jul '19

18 Jul '19

Hi Team, Is it supported for window OS. If Yes, please let me know more details about installation. Also let me know should I integrate with our product freely to support key management. Since it look like open source product. Thanks, Rayudu

2 1

OpenInfra Days Vietnam 2019 (Hanoi) - Call For Presentations
by Trinh Nguyen 12 Jul '19

12 Jul '19

Hello, Hope you're doing well :) The OpenInfra Days Vietnam 2019 [1] is looking for speakers in many different topics (e.g., container, CI, deployment, edge computing, etc.). If you would love to have a taste of Hanoi, the capital of Vietnam, please join us this one-day event and submit your presentation [2]. *- Date:* 24 AUGUST 2019 *- Location:* INTERCONTINENTAL HANOI LANDMARK72, HANOI, VIETNAM Especially this time, we're honored to have the Upstream Institute Training [3] hosted by the OpenStack Foundation on the next day (25 August 2019). [1] http://day.vietopeninfra.org/ [2] https://forms.gle/iiRBxxyRv1mGFbgi7 [3] https://docs.openstack.org/upstream-training/upstream-training-content.html See you in Hanoi! Bests, On behalf of the VietOpenInfra Group. -- *Trinh Nguyen* *www.edlab.xyz <https://www.edlab.xyz>*

2 2

[tc] Assuming control of GitHub organizations
by Jim Rollenhagen 10 Jul '19

10 Jul '19

Hi, The opendev team reached out to me about handing off administrative access of the "openstack" and related organizations on GitHub. They think it would be best if the TC took control of that, or at least took control of delegating that access. In general, the goal here is to support OpenStack's presence and visibility on GitHub. Per Jim Blair: > In the long run, this shouldn't entail a lot of work, generally creating new > repos on GitHub to accept mirroring from opendev systems, performing renames, > handling transfer requests when repos move out of the openstack namespace, > setting and updating descriptions, and curating the list of pinned > repositories. In the short term, we have some archiving and moving to do.[0] Do TC members want to manage this, or should we delegate? One thing to figure out is how to grant that access. The opendev team uses a shared account with two-factor authentication provided by a shared shell account. This mitigates accidental pushes or settings changes when an admin is using their usual GitHub account. The TC (or its delegates) probably doesn't have a shared shell account to do this with. Some options: * each admin creates a second GitHub account for this purpose use a shared * account without 2FA use a shared account with 2FA, share the one time secret * with everyone to configure their own token generator use personal accounts * but be very careful Thoughts on these options? [0] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006829.html // jim

8 15

[nova][ironic] Lock-related performance issue with update_resources periodic job
by Jason Anderson 09 Jul '19

09 Jul '19

Hey OpenStackers, I work on a cloud that allows users to reserve and provision bare metal instances with Ironic. We recently performed a long-overdue upgrade of our core components, all the way from Ocata up through Rocky. During this, we noticed that instance build requests were taking 4-5x (!!) as long as before. We have two deployments, one with ~150 bare metal nodes, and another with ~300. These are each managed by one nova-compute process running the Ironic driver. After investigation, the root cause appeared to be contention between the update_resources periodic task and the instance claim step. There is one semaphore "compute_resources" that is used to control every access within the resource_tracker. In our case, what was happening was the update_resources job, which runs every minute by default, was constantly queuing up accesses to this semaphore, because each hypervisor is updated independently, in series. This meant that, for us, each Ironic node was being processed and was holding the semaphore during its update (which took about 2-5 seconds in practice.) Multiply this by 150 and our update task was running constantly. Because an instance claim also needs to access this semaphore, this led to instances getting stuck in the "Build" state, after scheduling, for tens of minutes on average. There seemed to be some probabilistic effect here, which I hypothesize is related to the locking mechanism not using a "fair" lock (first-come, first-served) by default. Our fix was to drastically increase the interval this task runs at--from every 1 minute to every 12 hours. We only provision bare metal, so my rationale was that the periodic full resource sync was less important and mostly helpful for fixing weird things where somehow Placement's state got out of sync with Nova's somehow. I'm wondering, after all this, if it makes sense to rethink this one-semaphore thing, and instead create a per-hypervisor semaphore when doing the resource syncing. I can't think of a reason why the entire set of hypervisors needs to be considered as a whole when doing this task, but I could very well be missing something. TL;DR: if you have one nova-compute process managing lots of Ironic hypervisors, consider tweaking the update_resources_interval to a higher value, especially if you're seeing instances stuck in the Build state for a while. Cheers, Jason Anderson Cloud Computing Software Developer Consortium for Advanced Science and Engineering, The University of Chicago Mathematics & Computer Science Division, Argonne National Laboratory

5 11

[keystone] Virtual Midcycle Planning
by Colleen Murphy 08 Jul '19

08 Jul '19

Hi team, As discussed in today's meeting, we will be having a virtual midcycle some time around milestone 2. We'll do two days with one three-hour session (with breaks) each day. We will do this over a video conference session, details of how to join will follow closer to the event. I've started a brainstorming etherpad: https://etherpad.openstack.org/p/keystone-train-midcycle-topics Please add discussion topics or hacking ideas to the etherpad and I will try to sort them. We need to decide on when exactly to hold the midcycle. I've created a doodle poll: https://doodle.com/poll/wr7ct4uhpw82sysg Please select times and days that you're available and then we'll try to schedule two back-to-back days (or at least two days in the same week) for the midcycle. Let me know if you have any questions or concerns. Colleen

1 1

[qa][ptg][nova][cinder][keystone][neutron][glance][swift][placement] How to make integrated-gate testing (tempest-full) more stable and fast
by Ghanshyam Mann 07 Jul '19

07 Jul '19

Current integrated-gate jobs (tempest-full) is not so stable for various bugs specially timeout. We tried to improve it via filtering the slow tests in the separate tempest-slow job but the situation has not been improved much. We talked about the Ideas to make it more stable and fast for projects especially when failure is not related to each project. We are planning to split the integrated-gate template (only tempest-full job as first step) per related services. Idea: - Run only dependent service tests on project gate. - Tempest gate will keep running all the services tests as the integrated gate at a centeralized place without any change in the current job. - Each project can run the below mentioned template. - All below template will be defined and maintained by QA team. I would like to know each 6 services which run integrated-gate jobs 1."Integrated-gate-networking" (job to run on neutron gate) Tests to run in this template: neutron APIs , nova APIs, keystone APIs ? All scenario currently running in tempest-full in the same way ( means non-slow and in serial) Improvement for neutron gate: exlcude the cinder API tests, glance API tests, swift API tests, 2."Integrated-gate-storage" (job to run on cinder gate, glance gate) Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs, Nova APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial) Improvement for cinder, glance gate: excluded the neutron APIs tests, Keystone APIs tests 3. "Integrated-gate-object-storage" (job to run on swift gate) Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial) Improvement for swift gate: excluded the neutron APIs tests, - Keystone APIs tests, - Nova APIs tests. Note: swift does not run integrated-gate as of now. 4. "Integrated-gate-compute" (job to run on Nova gate) tests to run is : Nova APIs, Cinder APIs , Glance APIs ?, neutron APIs and All scenario currently running in tempest-full in same way ( means non-slow and in serial) Improvement for Nova gate: excluded the swift APIs tests(not running in current job but in future, it might), Keystone API tests. 5. "Integrated-gate-identity" (job to run on keystone gate) Tests to run is : all as all project use keystone, we might need to run all tests as it is running in integrated-gate. But does keystone is being unsed differently by all services? if no then, is it enough to run only single service tests say Nova or neutron ? 6. "Integrated-gate-placement" (job to run on placement gate) Tests to run in this template: Nova APIs tests, Neutron APIs tests + scenario tests + any new service depends on placement APIs Improvement for placement gate: excluded the glance APIs tests, cinder APIs tests, swift APIs tests, keystone APIs tests Thoughts on this approach? The important point is we must not lose the coverage of integrated testing per project. So I would like to get each project view if we are missing any dependency (proposed tests removal) in above proposed templates. - https://etherpad.openstack.org/p/qa-train-ptg -gmann

7 11

[masakari] Run masakari-hostmonitor into Docker container
by gaetan.trellu＠incloudus.com 06 Jul '19

06 Jul '19

Hi, I'm integrating Masakari into Kolla and Kolla Ansible projects but I'm facing an issue related to masakari-hostmonitor. Based on masakari-monitors code[1], "systemctl status" command is used to check if pacemaker, pacemaker-remote and corosync are running. Having systemd running into Docker container is not the best solution. Does any of you has been able to run masakari-monitor into Docker container ? Thanks for your help. Gaëtan - [1] https://github.com/openstack/masakari-monitors/blob/26d558333d9731ca06da09b…

3 2

[placement] db query analysis
by Chris Dent 05 Jul '19

05 Jul '19

One of the queries that has come up recently with placement performance is whether there may be opportunities to gain some improvement by making fewer queries to the db. That is: * are there redundant queries * are there places where data is gathered in multiple queries that could be in one (or at least fewer) * are there queries that are not doing what we think to that end I've done some analysis of logs produced when [placement_database]/connection_debug is set to 50 (which dumps SQL queries to the INFO log). The collection of queries made during a single request to GET /allocation_candidates is at http://paste.openstack.org/show/753183/ The data set is a single resource resource provider of the same form as used in the placement-perfload job (where 1000 providers are used). Only 1 is used in this case as several queries use 'IN' statements that list all the resource provider ids currently in play, and that gets dumped to the log making it inscrutable. I've noted in the paste where this happens. Each block of SQL is associated with the method that calls it. The queries are in the order they happen. One query that happens three times (once for each resource class requested) is listed once. Observations: * The way we use IN could be improved using a bindparam: https://docs.sqlalchemy.org/en/13/core/sqlelement.html?highlight=expanding%… * That we use IN in that fashion at all, where we are carrying lists of rp ids around and making multiple queries, instead of one giant one, might be an area worth exploring. * There are a couple of places where get get a trait id (via name) in a separate query from using the trait id. * What can you see? Please have a look to see if anything looks odd, wrong, etc. Basically what we're after is trying to find things that violate our expectations. Note that this is just one of several paths through the database. When there are sharing or nested providers things change. I didn't bother to do a more complex set of queries at this time as it seemed starting simple would help us tease out how best to communicate these sorts of things. Related to that, I've started working on a nested-perfload at https://review.opendev.org/665695 -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent

2 2

[kolla] Proposing yoctozepto as core
by Mark Goddard 05 Jul '19

05 Jul '19

Hi, I would like to propose adding Radosław Piliszek (yoctozepto) to kolla-core and kolla-ansible-core. While he has only recently started working upstream in the project, I feel that he has made some valuable contributions already, particularly around improving and maintaining CI. His reviews generally provide useful feedback, sometimes in advance of Zuul! Core team - please vote, and consider this my +1. I will keep this vote open for a week or until all cores have responded. Cheers, Mark

5 5

[kolla][kayobe] vote: kayobe as a kolla deliverable
by Mark Goddard 05 Jul '19

05 Jul '19

Hi, In the most recent kolla meeting [1] we discussed the possibility of kayobe becoming a deliverable of the kolla project. This follows on from discussion at the PTG and then on here [3]. The two options discussed are: 1. become a deliverable of the Kolla project 2. become an official top level OpenStack project There has been some positive feedback about option 1 and no negative feedback that I am aware of. I would therefore like to ask the kolla community to vote on whether to include kayobe as a deliverable of the kolla project. The electorate is the kolla-core and kolla-ansible core teams, excluding me. The opinion of others in the community is also welcome. If you have questions or feedback, please respond to this email. Once you have made a decision, please respond with your answer to the following question: "Should kayobe become a deliverable of the kolla project?" (yes/no) Thanks, Mark [1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.l… [2] https://etherpad.openstack.org/p/kolla-train-ptg [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006901.html

4 6

[tc][ops] reviving osops- repos
by Mohammed Naser 02 Jul '19

02 Jul '19

Hi everyone, So we've realized that we have a few operational tools that we keep around that live all over the place and it would probably be the best for the community to have access to them. I did a bit of research and it looks like the openstack/osops-* repos used to exist between operators to maintain things like this, however, they've gone pretty abandoned over time. I'd like to pick up this effort and as part of doing that, I'd like to move them into a GitHub organization and rename them out of x/ to make them not seem so .. 'dead'. I've requested to move x/osops-* to openstack-operators/* in GitHub so that I can setup the appropriate mirroring in post pipeline (and then propose a patch to rename things inside Gerrit as well). There was concern around the idea of using 'openstack' as it's a copyrighted term and a TC discussion was brought up as an idea which makes sense. I'd like to ask if everyone else feels like it's okay so move things into that org and host them there. Thanks, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

9 15