June 2019 - openstack-discuss - lists.openstack.org

Magnum Kubernetes openstack-cloud-controller-manager unable not resolve master node by DNS
by Pawel Konczalski 31 Jul '19

31 Jul '19

Hi, i try to deploy a Kubernetes cluster with OpenStack Magnum but the openstack-cloud-controller-manager pod fails to resolve the master node hostname. Does magnum require further parameter to configure the DNS names of the master and minions? DNS resolution in the VMs works fine. Currently there is no Designate installed in the OpenStack setup. openstack coe cluster template create kubernetes-cluster-template1 \ --image Fedora-AtomicHost-29-20190429.0.x86_64 \ --external-network public \ --dns-nameserver 8.8.8.8 \ --master-flavor m1.kubernetes \ --flavor m1.kubernetes \ --coe kubernetes \ --volume-driver cinder \ --network-driver flannel \ --docker-volume-size 25 openstack coe cluster create kubernetes-cluster1 \ --cluster-template kubernetes-cluster-template1 \ --master-count 1 \ --node-count 2 \ --keypair mykey # kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE kube-system coredns-78df4bf8ff-mjp2c 0/1 Pending 0 36m <none> <none> <none> kube-system heapster-74f98f6489-tgtzl 0/1 Pending 0 36m <none> <none> <none> kube-system kube-dns-autoscaler-986c49747-wrvz4 0/1 Pending 0 36m <none> <none> <none> kube-system kubernetes-dashboard-54cb7b5997-sk5pj 0/1 Pending 0 36m <none> <none> <none> kube-system openstack-cloud-controller-manager-dgk64 0/1 CrashLoopBackOff 11 36m <none> kubernetes-cluster1-vulg5fz6hg2n-master-0 <none> # kubectl -n kube-system logs openstack-cloud-controller-manager-dgk64 Error from server: Get https://kubernetes-cluster1-vulg5fz6hg2n-master-0:10250/containerLogs/kube-…: dial tcp: lookup kubernetes-cluster1-vulg5fz6hg2n-master-0 on 8.8.8.8:53: no such host BR Pawel

2 3

[forum][sdk][nova] Closing compute API feature gaps in the openstack CLI - recap
by Matt Riedemann 29 Jul '19

29 Jul '19

I wanted to give a quick recap of this Forum session for those that couldn't attend and also find owners. Please reply to this thread if you'd like to sign up for any specific item. The etherpad [1] has the details. To restate the goal: "Identify the major issues and functional gaps (up through Mitaka 2.25) and prioritize which to work on and what the commands should do." We spent the majority of the time talking about existing issues with compute API functionality in openstack CLI, primarily boot-from-volume, live migration and lack of evacuate support (evacuate as in rebuild on a new target host because the source host is dead, not drain a host with live migrations [2]). We then talked through some of the microversion gaps and picked a few to focus on. Based on that, the agreements and priorities are: **High Priority** 1. Make the boot-from-volume experience better by: a) Support type=image for the --block-device-mapping option. b) Add a --boot-from-volume option which will translate to a root --block-device-mapping using the provided --image value (nova will create the root volume under the covers). Owner: TBD (on either) 2. Fix the "openstack server migrate" command We're going to deprecate the --live option and add a new --live-migration option and a --host option. The --host option can be used for requesting a target host for cold migration (omit the --live/--live-migration option for that). Then in a major release we'll drop the --live option and intentionally not add a --force option (since we don't want to support forcing a target host and bypassing the scheduler). Owner: TBD (I would split the 2.56 cold migration --host support from the new --live-migration option review-wise) **Medium Priority** Start modeling migration resources in the openstack CLI, specifically for microversions 2.22-2.24, but note that the GET /os-migrations API is available since 2.1 (so that's probably easiest to add first). The idea is to have a new command resource like: openstack compute migration (list|delete|set) [--server <server_id>] Owner: TBD (again this is a series of changes) **Low Priority** Closing other feature gaps can be done on an as-needed basis as we've been doing today. Sean Mooney is working on adding evacuate support, and there are patches in flight (see [3]) for other microversion-specific features. I would like to figure out how to highlight these to the OSC core team on a more regular basis, but we didn't really talk about that. I've been trying to be a type of liaison for these patches and go over them before the core team tries to review them to make sure they match the API properly and are well documented. Does the OSC core team have any suggestions on how I can better socialize what I think is ready for core team review? [1] https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps [2] http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-co… [3] https://etherpad.openstack.org/p/compute-api-microversion-gap-in-osc -- Thanks, Matt

2 2

[dev][magnum] Magnum event notifications not having relevant content related to clusters
by Emnauel Andrecut 23 Jul '19

23 Jul '19

I'm writing about Magnum event notifications that do not contain any relevant information about the related Cluster (missing cluster id, other attributes..) As I've seen the PyCADF library receives no targetId and so the id (and mostly any other id) is randomly generated. Do you plan to add more information about the cluster on create success/pending events. Or would you accept a patch that does that?

3 2

[all] [ptls] [tc] [nova] [neutron] Volunteers that know TeX for PDF community goal
by Alexandra Settle 19 Jul '19

19 Jul '19

Hi all, The work for the Train community goal - PDF support for project docs - is well underway. [1] Now, we're looking for volunteers to help test the implementation. We'll need someone to help build the docs into PDFs and determine things we can fix through tweaks to our docs, or if they're bugs in Sphinx. AKA: We need a troubleshoot artist. If you can volunteer, please add yourself to the wiki table here [2]. I've added neutron and nova specifically here as we need someone who is familiar with the project and it's dependencies to help us get that setup. Any questions? Reach out. Cheers, Alex [1] https://review.opendev.org/#/q/topic:build-pdf-docs+(status:open+OR+status:… [2] https://wiki.openstack.org/wiki/Documentation#PDF_for_Project_Docs_-_Commun…

8 12

Barbican support for Window
by Garaga Rayudu 18 Jul '19

18 Jul '19

Hi Team, Is it supported for window OS. If Yes, please let me know more details about installation. Also let me know should I integrate with our product freely to support key management. Since it look like open source product. Thanks, Rayudu

2 1

OpenInfra Days Vietnam 2019 (Hanoi) - Call For Presentations
by Trinh Nguyen 13 Jul '19

13 Jul '19

Hello, Hope you're doing well :) The OpenInfra Days Vietnam 2019 [1] is looking for speakers in many different topics (e.g., container, CI, deployment, edge computing, etc.). If you would love to have a taste of Hanoi, the capital of Vietnam, please join us this one-day event and submit your presentation [2]. *- Date:* 24 AUGUST 2019 *- Location:* INTERCONTINENTAL HANOI LANDMARK72, HANOI, VIETNAM Especially this time, we're honored to have the Upstream Institute Training [3] hosted by the OpenStack Foundation on the next day (25 August 2019). [1] http://day.vietopeninfra.org/ [2] https://forms.gle/iiRBxxyRv1mGFbgi7 [3] https://docs.openstack.org/upstream-training/upstream-training-content.html See you in Hanoi! Bests, On behalf of the VietOpenInfra Group. -- *Trinh Nguyen* *www.edlab.xyz <https://www.edlab.xyz>*

2 2

[tc] Assuming control of GitHub organizations
by Jim Rollenhagen 10 Jul '19

10 Jul '19

Hi, The opendev team reached out to me about handing off administrative access of the "openstack" and related organizations on GitHub. They think it would be best if the TC took control of that, or at least took control of delegating that access. In general, the goal here is to support OpenStack's presence and visibility on GitHub. Per Jim Blair: > In the long run, this shouldn't entail a lot of work, generally creating new > repos on GitHub to accept mirroring from opendev systems, performing renames, > handling transfer requests when repos move out of the openstack namespace, > setting and updating descriptions, and curating the list of pinned > repositories. In the short term, we have some archiving and moving to do.[0] Do TC members want to manage this, or should we delegate? One thing to figure out is how to grant that access. The opendev team uses a shared account with two-factor authentication provided by a shared shell account. This mitigates accidental pushes or settings changes when an admin is using their usual GitHub account. The TC (or its delegates) probably doesn't have a shared shell account to do this with. Some options: * each admin creates a second GitHub account for this purpose use a shared * account without 2FA use a shared account with 2FA, share the one time secret * with everyone to configure their own token generator use personal accounts * but be very careful Thoughts on these options? [0] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006829.html // jim

8 15

[nova][ironic] Lock-related performance issue with update_resources periodic job
by Jason Anderson 09 Jul '19

09 Jul '19

Hey OpenStackers, I work on a cloud that allows users to reserve and provision bare metal instances with Ironic. We recently performed a long-overdue upgrade of our core components, all the way from Ocata up through Rocky. During this, we noticed that instance build requests were taking 4-5x (!!) as long as before. We have two deployments, one with ~150 bare metal nodes, and another with ~300. These are each managed by one nova-compute process running the Ironic driver. After investigation, the root cause appeared to be contention between the update_resources periodic task and the instance claim step. There is one semaphore "compute_resources" that is used to control every access within the resource_tracker. In our case, what was happening was the update_resources job, which runs every minute by default, was constantly queuing up accesses to this semaphore, because each hypervisor is updated independently, in series. This meant that, for us, each Ironic node was being processed and was holding the semaphore during its update (which took about 2-5 seconds in practice.) Multiply this by 150 and our update task was running constantly. Because an instance claim also needs to access this semaphore, this led to instances getting stuck in the "Build" state, after scheduling, for tens of minutes on average. There seemed to be some probabilistic effect here, which I hypothesize is related to the locking mechanism not using a "fair" lock (first-come, first-served) by default. Our fix was to drastically increase the interval this task runs at--from every 1 minute to every 12 hours. We only provision bare metal, so my rationale was that the periodic full resource sync was less important and mostly helpful for fixing weird things where somehow Placement's state got out of sync with Nova's somehow. I'm wondering, after all this, if it makes sense to rethink this one-semaphore thing, and instead create a per-hypervisor semaphore when doing the resource syncing. I can't think of a reason why the entire set of hypervisors needs to be considered as a whole when doing this task, but I could very well be missing something. TL;DR: if you have one nova-compute process managing lots of Ironic hypervisors, consider tweaking the update_resources_interval to a higher value, especially if you're seeing instances stuck in the Build state for a while. Cheers, Jason Anderson Cloud Computing Software Developer Consortium for Advanced Science and Engineering, The University of Chicago Mathematics & Computer Science Division, Argonne National Laboratory

5 11

[keystone] Virtual Midcycle Planning
by Colleen Murphy 08 Jul '19

08 Jul '19

Hi team, As discussed in today's meeting, we will be having a virtual midcycle some time around milestone 2. We'll do two days with one three-hour session (with breaks) each day. We will do this over a video conference session, details of how to join will follow closer to the event. I've started a brainstorming etherpad: https://etherpad.openstack.org/p/keystone-train-midcycle-topics Please add discussion topics or hacking ideas to the etherpad and I will try to sort them. We need to decide on when exactly to hold the midcycle. I've created a doodle poll: https://doodle.com/poll/wr7ct4uhpw82sysg Please select times and days that you're available and then we'll try to schedule two back-to-back days (or at least two days in the same week) for the midcycle. Let me know if you have any questions or concerns. Colleen

1 1

[qa][ptg][nova][cinder][keystone][neutron][glance][swift][placement] How to make integrated-gate testing (tempest-full) more stable and fast
by Ghanshyam Mann 07 Jul '19

07 Jul '19

Current integrated-gate jobs (tempest-full) is not so stable for various bugs specially timeout. We tried to improve it via filtering the slow tests in the separate tempest-slow job but the situation has not been improved much. We talked about the Ideas to make it more stable and fast for projects especially when failure is not related to each project. We are planning to split the integrated-gate template (only tempest-full job as first step) per related services. Idea: - Run only dependent service tests on project gate. - Tempest gate will keep running all the services tests as the integrated gate at a centeralized place without any change in the current job. - Each project can run the below mentioned template. - All below template will be defined and maintained by QA team. I would like to know each 6 services which run integrated-gate jobs 1."Integrated-gate-networking" (job to run on neutron gate) Tests to run in this template: neutron APIs , nova APIs, keystone APIs ? All scenario currently running in tempest-full in the same way ( means non-slow and in serial) Improvement for neutron gate: exlcude the cinder API tests, glance API tests, swift API tests, 2."Integrated-gate-storage" (job to run on cinder gate, glance gate) Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs, Nova APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial) Improvement for cinder, glance gate: excluded the neutron APIs tests, Keystone APIs tests 3. "Integrated-gate-object-storage" (job to run on swift gate) Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial) Improvement for swift gate: excluded the neutron APIs tests, - Keystone APIs tests, - Nova APIs tests. Note: swift does not run integrated-gate as of now. 4. "Integrated-gate-compute" (job to run on Nova gate) tests to run is : Nova APIs, Cinder APIs , Glance APIs ?, neutron APIs and All scenario currently running in tempest-full in same way ( means non-slow and in serial) Improvement for Nova gate: excluded the swift APIs tests(not running in current job but in future, it might), Keystone API tests. 5. "Integrated-gate-identity" (job to run on keystone gate) Tests to run is : all as all project use keystone, we might need to run all tests as it is running in integrated-gate. But does keystone is being unsed differently by all services? if no then, is it enough to run only single service tests say Nova or neutron ? 6. "Integrated-gate-placement" (job to run on placement gate) Tests to run in this template: Nova APIs tests, Neutron APIs tests + scenario tests + any new service depends on placement APIs Improvement for placement gate: excluded the glance APIs tests, cinder APIs tests, swift APIs tests, keystone APIs tests Thoughts on this approach? The important point is we must not lose the coverage of integrated testing per project. So I would like to get each project view if we are missing any dependency (proposed tests removal) in above proposed templates. - https://etherpad.openstack.org/p/qa-train-ptg -gmann

7 11