February 2019 - openstack-discuss

[kolla][magnum] Cluster creation failed due to "Waiting for Kubernetes API..."
by Giuseppe Sannino 30 May '19

30 May '19

Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled --------------- VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out" I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5 Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE) Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver. [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1. May I ask for an help on this ? Many thanks /Giuseppe

5 11

Scuba Diving at the Denver Summit
by Monty Taylor 23 Apr '19

23 Apr '19

Hey everybody! tl;dr - Let me know if you'd like to go diving in the Rocky Mountains. Of all of the places we've had a Summit where one might reasonably expect there to be a possibility of Scuba Diving, the Denver Summit is probably REALLY low on the list. But it turns out that there is a Dive Shop that does dives at the Denver Aquarium [0] (with all the fish and sharks and whatnot) The Denver Aquarium is also very close to the Convention Center. I'm going to see about putting together a private group dive event for those of us who think that sounds like a fun idea. Due to cross-contamination concerns for the health and safety of the fish, the dive shop provides all equipment - so you don't have to bring gear. They will allow bringing your own mask - and I'm assuming if a well-scrubbed mask is ok then a well-scrubbed dive computer would be ok - but I'll be checking with them when I call to arrange things. If you are Open Water certified and would like to join, please send me a response back (reply directly to me rather than the list is fine) so I can judge general interest level when talking to the lovely humans. If you are **NOT** Open Water certified but are thinking to yourself "wow, that sounds like fun" - the dive shop does certification dives at the aquarium on the weekends. That's a bit out of scope for me to arrange, but if you want to show up in Denver early and do an Open Water class then join us, that's awesome. You can also get certified at home before coming ... but you'll need an Open Water certification to join us. BUT WAIT - THERE'S MORE! There is another interesting diving opportunity in the general Mountain West area - although not immediately **close** to Denver - that Sandy and I are going to be going to before the Summit - Homestead Crater. We're arranging to meet a Scuba instructor there to do our second Altitude cert dive, and also to do a 2-dive "Hot Springs Diver" specialty. We'd be happy to have folks join us. Homestead Crater [1] is outside of Salt Lake City. It's a hot spring - the water is 96F/35.5C and it's also an altitude dive. If people are interested, we can meet at the crater on Thursday morning pre-summit, dive the crater, then drive to Denver on Friday/Saturday. ** If only a small number of people are interested in any of this - we'll likely just do the Denver dives the weekend before during one of the normally scheduled dive times instead of arranging a group outing, and shift Homestead, if anyone is interested, a few days earlier ** I will not have space enough in my car to get you from the crater to Denver, so if you want to fly in to SLC ahead of time and meet there, awesome. Unlike Denver, the crater will require you have equipment, including scuba tanks. For anyone interested in coming, there are dive shops in Salt Lake where you can rent gear. There is a dive shop at the crater to do air fills. Anyway - if anyone is interested in some or all of these things, let me know this week so I can get a headcount and then we can talk more off-list about logistics and figure out how to deal with things for people coming from far off (carpools, equipment, etc) Monty PS. If any of you aren't already using Subsurface for dive logging, I'll be happy to help you get set up. [0] https://divedowntown.com/ [1] https://homesteadresort.com/utah-resort-things-to-do/homestead-crater/

4 5

Call for help! 'U' Release name Mandarin speakers
by Tony Breeds 22 Apr '19

22 Apr '19

Hello folks, Now that we know when and where the 'U' summit will be[1] we can pick the name for the U release. I'm happy to act as coordinator but given the location I'd really like a native speaker (or 2) to help with assessing the names against the criteria, defining the geographic region and translating the civs POLL when we get there. Yours Tony. [1] https://www.openstack.org/summit/shanghai-2019

9 13

Proposal to move cinder backup tests out of the integrated gate
by Matt Riedemann 11 Apr '19

11 Apr '19

I wanted to send this separate from the latest gate status update [1] since it's primarily about latent cinder bugs causing failures in the gate for which no one is really investigating. Running down our tracked gate bugs [2] there are several related to cinder-backup testing: * http://status.openstack.org/elastic-recheck/#1483434 * http://status.openstack.org/elastic-recheck/#1745168 * http://status.openstack.org/elastic-recheck/#1739482 * http://status.openstack.org/elastic-recheck/#1635643 All of those bugs were reported a long time ago. I've done some investigation into them (at least at the time of reporting) and some are simply due to cinder-api using synchronous RPC calls to cinder-volume (or cinder-backup) and that doesn't scale. This bug isn't a backup issue, but it's definitely related to using RPC call rather than cast: http://status.openstack.org/elastic-recheck/#1763712 Regarding the backup tests specifically, I don't see a reason why they need to be run in the integrated gate jobs, e.g. tempest-full(-py3). They don't involve other services, so in my opinion we should move the backup tests to a separate job which only runs on cinder changes to alleviate these latent bugs failing jobs for unrelated changes and resetting the entire gate. I would need someone from the cinder team that is more involved in knowing what their job setup looks like to identify a candidate job for these tests if this is something everyone can agree on doing. [1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000867… [2] http://status.openstack.org/elastic-recheck/ -- Thanks, Matt

6 12

[horizon] Unused xstatic-* projects retirement
by Ivan Kolodyazhny 03 Apr '19

03 Apr '19

Hi team, There are some xstatic packages which I didn't start to use in Horizon or plugins. We didn't do any release of them. During the last meeting [1] we agreed to mark them as retired. I'll start retired procedure [2] today. If you're going to use them, please let me know. The list of the projects to be retired: - xstatic-angular-ui-router - xstatic-bootstrap-datepicker - xstatic-hogan - xstatic-jquery-migrate - xstatic-jquery.quicksearch - xstatic-jquery.tablesorter - xstatic-rickshaw - xstatic-spin - xstatic-vis [1] http://eavesdrop.openstack.org/meetings/horizon/2019/horizon.2019-01-09-15.… Regards, Ivan Kolodyazhny, http://blog.e0ne.info/

3 4

[stable][kolla] EOL kolla ocata branch
by Eduardo Gonzalez 03 Apr '19

03 Apr '19

Hi stable team, Kolla project is ready to tag EOL the stable/ocata branch. We would like to tag the following repositories as EOL: - openstack/kolla - openstack/kolla-ansible Let me know if need anything else. Regards

4 5

[first contact] Meeting Time Moving!
by Kendall Nelson 30 Mar '19

30 Mar '19

Hello :) Since two of our regular meeting attendees are moving/have moved to very different timezones, here is a poll to pick a new time to meet! We also won't actually enact the new time until April-ish but if you have a preference on if we want to move it for the first meeting in April or the second, please voice your opinions here :) -Kendall (diablo_rojo) [1] https://doodle.com/poll/h5m5n6za9hbiv9pr

3 5

[requirements][mistral] Add yamlloader to global requirements
by Oleg Ovcharuk 28 Mar '19

28 Mar '19

Hi! Can you please add yamlloader library to global requirements? https://pypi.org/project/yamlloader/ It provides ability to preserve key order in dicts, it supports either python 2.7 and python 3.x, it provides better performance than built-in functions. Thank you.

8 13

[placement] [translation] translating exceptions
by Chris Dent 25 Mar '19

25 Mar '19

In placement we've been experimenting with removing oslo_versionedobjects. There's neither RPC nor rolling upgrades in placement so it is effectively overkill and the experiment has revealed that removing it also improves performance. The removal of OVO allows us to remove a lot of transitive dependencies. See https://review.openstack.org/#/c/636807/ where the lower-constraints.txt file is updated to reflect the changes. After those changes the biggest single package remaining in the test virtualenvs is Babel (included via oslo.i18n), weighing in at a mighty 26M. Placement is currently set to enable translation of exception messages, especially those that will end up in API responses. It has no other UI to speak of, so we've begun to wonder about getting rid of translation entirely. I asked about this in the TC office hours[1], which generated some useful but not entirely conclusive discussion. One way to summarize that discussion is that it might be okay to not translate the error messages, because it is unlikely they will be a priority for translators who are more oriented towards docs and UI, and that since no translation has yet happened, if there were ever going to be a time to turn off translation, now would be the time. It is additionally okay because placement is striving to follow the errors guideline [2] and use error.code in error responses, which exist at least in part to make googling for "what do I do when I get this error" a bit easier. That desire to google is also a good reason to keep the error responses in one language. However: We shouldn't do this if people need or want the error responses to be translated. Thus this message. Do people have opinions on this? Is it okay to not translate error responses in an API-only service like placement? Thanks. [1] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2019… [2] https://specs.openstack.org/openstack/api-wg/guidelines/errors.html -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

5 7

[dev] [all] [ptl] Migrating legacy jobs to Bionic (Ubuntu LTS 18.04)
by Ghanshyam Mann 25 Mar '19

25 Mar '19

Hi Everyone, During Dec/Jan month, we have migrated the devstack jobs (zuulv3 native jobs) from Xenial to Bionic. [1]. But that did not move all gate job to Bionic as there are a large number of jobs are still the legacy job. All the legacy job still use Xenial as nodeset. As per the decided runtime for Stein, we need to test everything on OpenStack CI/CD on Bionic - https://governance.openstack.org/tc/reference/runtimes/stein.html This is something we discussed TC meeting as one of the Stein items to finish in openstack community[2]. I am starting this ML thread to coordinate the work to move all the legacy jobs to Bionic. I have created the Etherpad to include more details and status check for each project gate- https://etherpad.openstack.org/p/legacy-job-bionic The approach is the same as we did for previous migration for devstack jobs. 1. Push patch which migrates the legacy base jobs to bionic - 1. https://review.openstack.org/#/c/639096/ 2. Each Project team, add the testing patch with Depends-on on base patch (example 639096) and confirm the status of their gate on etherpad - Example: https://review.openstack.org/#/c/639017 3. Project jobs not using base job as parent, need to migrate their legacy job to bionic by own. Please add yourself as your project volunteer in etherpad and update the status accordingly. I am tagging [ptl] in this thread so that they can assign someone from their team if no volunteer. Deadline: 1st April to merge the base legacy job to bionic. That gives around 1 month to test the jobs which I feel enough for each project. let me know if more time is needed, we can adjust the same. The goal is to finish this activity before Stein release. [1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000837… https://etherpad.openstack.org/p/devstack-bionic [2] http://eavesdrop.openstack.org/meetings/tc/2019/tc.2019-02-07-14.00.html -gmann

10 21