- openstack-discuss - lists.openstack.org

[stein][cinder][backup][swift] issue
by Ignazio Cassano 03 Jan '20

03 Jan '20

Hello All, I configured openstack stein on centos 7 witch ceph. Cinder works fine and object storage on ceph seems to work fine: I can clreate containers, volume etc ..... I configured cinder backup on swift (but swift is using ceph rados gateway) : backup_driver = cinder.backup.drivers.swift.SwiftBackupDriver swift_catalog_info = object-store:swift:publicURL backup_swift_enable_progress_timer = True #backup_swift_url = http://10.102.184.190:8080/v1/AUTH_ backup_swift_auth_url = http://10.102.184.190:5000/v3 backup_swift_auth = per_user backup_swift_auth_version = 1 backup_swift_user = admin backup_swift_user_domain = default #backup_swift_key = <None> #backup_swift_container = volumebackups backup_swift_object_size = 52428800 #backup_swift_project = <None> #backup_swift_project_domain = <None> backup_swift_retry_attempts = 3 backup_swift_retry_backoff = 2 backup_compression_algorithm = zlib If I run a backup as user admin, it creates a container named "volumebackups". If I run a backup as user demo and I do not specify a container name, it tires to write on volumebackups and gives some errors: ClientException: Container PUT failed: http://10.102.184.190:8080/swift/v1/AUTH_964f343cf5164028a803db91488bdb01/v… 409 Conflict BucketAlreadyExists Does it mean I cannot use the same containers name on differents projects ? My ceph.conf is configured for using keystone: [client.rgw.tst2-osctrl01] rgw_frontends = "civetweb port=10.102.184.190:8080" # Keystone information rgw keystone api version = 3 rgw keystone url = http://10.102.184.190:5000 rgw keystone admin user = admin rgw keystone admin password = password rgw keystone admin domain = default rgw keystone admin project = admin rgw swift account in url = true rgw keystone implicit tenants = true Any help, please ? Best Regards Ignazio

3 4

[openstack-ansible] strange execution delays
by Joe Topjian 02 Jan '20

02 Jan '20

Hello, I don't think this is an OSA issue per-se, but it's happening on infrastructure deployed with OSA. Using both the Rocky and Stein releases of OSA on Ubuntu 18.04, we're running into a strange a strange issue where the run times of Ansible will eventually take longer and longer. Digging into the issue more, we find that it's when Ansible executes a command within a container and the call to "su" triggers the delay. Over time, the delay increases more and more. The same behavior is seen if we SSH into containers. The only workaround we've been able to identify that works is to stop and start the containers. We have not been able to identify a root cause of this, nor any consistency to what containers this will happen on. Has anyone run into this? Thanks, Joe

2 5

Re: openstack-discuss Digest, Vol 15, Issue 1
by Soniya Vyas 02 Jan '20

02 Jan '20

> Message: 4 > Date: Tue, 31 Dec 2019 19:03:22 -0600 > From: Ghanshyam Mann <gmann(a)ghanshyammann.com> > To: "openstack-discuss" <openstack-discuss(a)lists.openstack.org> > Subject: [qa] QA Office hour new timing for 2020 > Message-ID: > <16f5ea0ef6f.1010065b175496.3744470588375359656(a)ghanshyammann.com> > Content-Type: text/plain; charset="UTF-8" > > Hello Everyone, > > We have few members who actively started contribution to QA from India as well as from Europe TZ. > The current office hour time is more convenient for CT and JST only and very difficult for members from India > Europe to join. > > I would like to adjust office hour timing to include all four TZs. To do that someone has to wake up early or stay > late at night :). I gave preference to new members and selected Tuesday 13.30 UTC [1]. which will be an early morning > for me and late-night in Tokyo. > > let me know if any objection or better suggestion. Accordingly, I will make the new time effective from 7th Jan. > > Also cancelling the 1st Jan office hour and wish you all a very happy new year. > > [1] https://www.timeanddate.com/worldclock/meetingdetails.html?year=2020&month=… Thanks a lot to whole QA team and Ghanshyam Mann for considering our timing issues. Looking forward to join QA Office hours meeting. Regards, Soniya Vyas

1 0

[manila] No IRC/Community meeting on 2nd January 2020
by Goutham Pacha Ravi 01 Jan '20

01 Jan '20

Hello Zorillas, Due to the holidays we're expecting multiple community members (myself included) to be unavailable for the weekly IRC meeting tomorrow (2nd January 2020, 15:00 UTC), so we'll skip it. Please add any agenda items to the next meeting (9th January 2020, 15:00 UTC) [1]. Happy New Year, and the rest of your holidays! Thanks, Goutham [1] https://wiki.openstack.org/wiki/Manila/Meetings

1 0

[qa] QA Office hour new timing for 2020
by Ghanshyam Mann 01 Jan '20

01 Jan '20

Hello Everyone, We have few members who actively started contribution to QA from India as well as from Europe TZ. The current office hour time is more convenient for CT and JST only and very difficult for members from India Europe to join. I would like to adjust office hour timing to include all four TZs. To do that someone has to wake up early or stay late at night :). I gave preference to new members and selected Tuesday 13.30 UTC [1]. which will be an early morning for me and late-night in Tokyo. let me know if any objection or better suggestion. Accordingly, I will make the new time effective from 7th Jan. Also cancelling the 1st Jan office hour and wish you all a very happy new year. [1] https://www.timeanddate.com/worldclock/meetingdetails.html?year=2020&month=… -gmann

1 0

[neutron] Slow floating IP activation when FWaaS v2 enabled
by Jason Anderson 31 Dec '19

31 Dec '19

Hi all, We recently started using the FWaaS feature (v2) in Neutron for our deployment. However, we noticed that when this feature is enabled, floating IP activation (the setting up of iptables rules that perform DNAT to the destination instance) takes a long time to happen. The effect is that the user has to wait for a while before the floating IP becomes active on an instance. I was wondering if anybody else has experienced this, and if there are any workarounds. I suspect there may be some queuing happening behind an iptables lock somewhere but I haven't been able to verify anything for sure yet. Thank you, -- Jason Anderson Chameleon DevOps Lead Consortium for Advanced Science and Engineering, The University of Chicago Mathematics & Computer Science Division, Argonne National Laboratory

1 0

[horizon][requirements] django-3 blocked by usage of django-babel
by Matthew Thode 31 Dec '19

31 Dec '19

it looks like django-babel is unmaintained since 2017, so perhaps it's time to look for a replacement. Looking through github issues, it looks like enmerkar is a possible replacement. If that's fine with horizon / django consumers we can start by adding enmerkar to global-requirements and upper-constraints. Thanks, -- Matthew Thode

4 9

[neutron] Bug deputy report week of December 23rd
by Miguel Lavalle 30 Dec '19

30 Dec '19

Hi, I was the bug deputy for the week of December 23rd. As expected for an end of year Holidays week, We had very little activity: Requested more information: - https://bugs.launchpad.net/neutron/+bug/1857422 neutron-keepalived-state-change and keeplived cannot be cleanup for those routers which is deleted during l3-agent died

2 1

Openstack Rocky to Train upgrade
by Md. Farhad Hasan Khan 30 Dec '19

30 Dec '19

Hi, Could you please help me how can I upgrade from Openstack Rocky version to Train. Thanks & B'Rgds, Rony

3 5

[devstack][ceilometer][tacker]ceilometer devstack installation fails
by Hiroyuki JO 25 Dec '19

25 Dec '19

Hi ceilometer team and all, After a patch[1] was merged, Tacker zuul check fails because ceilometer devstack installation error[2]. Do you have any work aroud of this problem? Addition, why devstack uses pip version 9.0.3 when installing python packages? The cause of our problem is installation error of conluent-kafka. But with latest pip (version 19), I was able to install the package. [1] https://opendev.org/openstack/requirements/commit/c99ab09bc44818b92fe86842f… [2] https://zuul.opendev.org/t/openstack/build/e6d04b35d16e4447901aba441721115a… Regards, 徐広幸 (Hiroyuki JO) Email: hiroyuki.jo.mt(a)hco.ntt.co.jp / Tel(direct): +81-422-59-7394 NTT Network Systems Laboratories

3 5

[cyborg] No weekly IRC meeting for next 2 weeks
by Nadathur, Sundar 24 Dec '19

24 Dec '19

There will be no weekly Cyborg IRC meeting on Dec 26 or Jan 2. Happy Holidays to all. Regards, Sundar

1 0

[neutron] bug deputy report of week 2019-12-16
by Bence Romsics 23 Dec '19

23 Dec '19

Hi Neutron Team, Here comes the deputy report of last week: High: * https://bugs.launchpad.net/neutron/+bug/1856600 Unit test jobs are failing with ImportError: cannot import name 'engine' from 'flake8' Gate unblocking fix by Brian merged already. Follow up proposed by Nate: https://review.opendev.org/699243 Medium: * https://bugs.launchpad.net/neutron/+bug/1856523 Sometimes instance can't get public keys due to cirros metadata request failur Occasional loss of metadata response. Follow up bug report for CirrOS by Maciej: https://github.com/cirros-dev/cirros/issues/8 and pull request on github: https://github.com/cirros-dev/cirros/pull/11 * https://bugs.launchpad.net/neutron/+bug/1856675 / https://bugs.launchpad.net/neutron/+bug/1856726 IPv6 Prefix Delegation do not work / IpamBackendMixin._gateway_ip_str is not reading correctly the subnet IP version Fix proposed by Rodolfo: https://review.opendev.org/699465 * https://bugs.launchpad.net/neutron/+bug/1856853 OVSInterfaceDriver.plug_new should catch pyroute2.netlink.exceptions.NetlinkErro Fix proposed by Rodoflo: https://review.opendev.org/699700 * https://bugs.launchpad.net/neutron/+bug/1857021 Wrong order of adding/deleting router ports may end up with not configured router port Fix proposed by Slawek: https://review.opendev.org/700011 Low: * https://bugs.launchpad.net/neutron/+bug/1857035 Add better description for "test_cleanup_network_namespaces_cleans_dhcp_and_l3_namespaces" in case of failure Low hanging fruit improvement to logging. Unassigned. * https://bugs.launchpad.net/neutron/+bug/1857047 openstack port set --no-fixed-ips removed only 1 fixed IP Unassigned. Undecided: * https://bugs.launchpad.net/neutron/+bug/1857016 Possible double row.delete() call in ACL code neutron fix by Terry in the gate: https://review.opendev.org/700007 the same change in neutron-ovn: https://review.opendev.org/700001 Maybe RFE: * https://bugs.launchpad.net/neutron/+bug/1856839 [L3] router processing time increase if there are large large set ports Possbile performance problem reported by Yulong (who is also the assignee). No gerrit change yet. Merry Christmas, Bence

1 0

[heat] Addressing the large patch backlog.
by David Peacock 23 Dec '19

23 Dec '19

Hi all, For those interested in the workings of the Heat project, I'd like to kick off a call to action. At the time of writing there are approximately 200 open patches against the core Heat project repo alone, not counting the other Heat repos. Recently I started going through and triaging the patches I'd consider "historical" with an arbitrary cut off for this definition of August 1st of this year. There are 148 patches which meet this definition, dating all the way back to 2015. I have gone through them all and placed them into a spreadsheet [0] which I'd invite you all to check. Provided is a link to the patch in question, initial upload date, last meaningful update date, primary author, and a high level summary of the patch. Additionally I've broken the patches down into three recommended states based on a high level first pass. *Abandon* 34 patches are candidates to be abandoned; they usually are of debatable utility, have significant outstanding concerns, or have no followup from the original developer in a very long time. In many cases, all of these conditions. *Without good reason or explanation from the original developer, these patches may ultimately be cleared out.* *Rebase + Merge* 38 patches are with a high level look in reasonably good shape, perform a stated goal, and may be trivial to core review and ultimately rebase and merge. *If you're the original developer or otherwise interested in these patches and wish to see them through the merge process, please rebase the patch.* *Research* 76 patches are sufficiently complex that they'll need a much closer look. Some of these patches are in a seemingly "finished" state, some are a way off. Some have unanswered concerns from core review and have been left dangling. *If you're the original developer or otherwise interested in working these patches through to completion, please do get involved.* When I started this little mission I wasn't quite sure what to expect. What I have found is that as much as there was anticipated cruft to clear out, there is a great deal of very good work lurking here, waiting to see the light of day, and it would be so good to see this work realised. :-) If you have anything to say, feel free to write back on list, and if you'd like to coordinate with me any efforts with these patches I can be found by email or on Freenode in the #heat channel; I'm dpeacock. Based on feedback of this idea, and indeed on each individual patch, I hope we can get this backlog under control, and harvest some of this excellent code! Thank you, David Peacock [0] https://ethercalc.openstack.org/b3qtqyhkg9g1 Please be mindful of accidental edits.

2 2

[kuryr] PTL on vacations
by Michał Dulko 23 Dec '19

23 Dec '19

Hi, I'm starting my vacations now and will be back on January 13th. Please direct any issues to the rest of the core team. Thanks, Michał

1 0

[tripleo] TripleO CI Summary: Sprint 40
by Rafael Folco 23 Dec '19

23 Dec '19

Greetings, The TripleO CI team has just completed Sprint 40 / Unified Sprint 19 (Nov 21 thru Dec 18). The following is a summary of completed work during this sprint cycle: - Created a PoC for 3rd-party testing jobs for podman and ceph-ansible. The 3rd-party jobs are now running and being triggered from github pull requests. - Closed-out work on the design enhancements to the promotion server. Tested the new promoter code and fixed issues related to manifests and quay.io. - Working w/ quay support re: performance and api issues. - Implemented component pipeline [3] running daily and promoting separated from the integration jobs. Job that promotes the component still under testing and fixing bugs. Component jobs have been also created in downstream. - Addressed issues and technical debt tasks in TripleO CI realm, including issues in the zuul reproducer. Next sprint starts Jan 3rd. The planned work for the next sprint [1] are still going to be defined, and may include component pipeline work and build out CentOS8 promotion and check jobs upstream. The Ruck and Rover for this sprint are TBD. Please direct questions or queries to them regarding CI status or issues in #tripleo, ideally to whomever has the ‘|ruck’ suffix on their nick. Ruck/rover notes to be tracked in etherpad [2]. Thanks, rfolco [1] https://tree.taiga.io/project/tripleo-ci-board/taskboard/unified-sprint-20 [2] https://etherpad.openstack.org/p/ruckroversprint20 [3] http://dashboard-ci.tripleo.org/d/UDA4H3aZk/component-pipeline?orgId=1

1 0

[kuryr] Multi-net spec 1.1 released
by Michał Dulko 23 Dec '19

23 Dec '19

Hi, I just wanted to give a heads up that version 1.1 of Multi-Network CRD Specification was just released. It might be worth to implement support for it in Kuryr at some point if there's something we could benefit from. Thanks, Michał

1 0

[octavia] Holiday Meeting Cancellations!
by Adam Harwell 21 Dec '19

21 Dec '19

Hey all! Octavia meetings will be cancelled for the next two weeks (Dec 25 and Jan 1), and will resume on Jan 8. See y'all then, and have a great holiday season! --Adam

1 0

[horizon] team meeting cancelled in the holiday season
by Akihiro Motoki 21 Dec '19

21 Dec '19

Hi, Horizon team cancel the team meetings at Dec 25, 2019 and Jan 1, 2020. See you all in new year and enjoy holidays! Thanks, Akihiro Motoki (irc: amotoki)

1 0

State of the Gate
by Matt Riedemann 20 Dec '19

20 Dec '19

Things are great! Surprise! I just wanted to let everyone know. Later! . . . . . Now that you've been tricked, on Halloween no less, I'm here to tell you that things suck right now. This is your periodic digest of issues. Grab some fun-sized candy bars and read on. I think right now we have three major issues. 1. http://status.openstack.org/elastic-recheck/index.html#1763070 This has resurfaced and I'm not sure why, nor do I think we ever had a great handle on what is causing this or how to work around it so if anyone has new ideas please chip in. 2. http://status.openstack.org/elastic-recheck/index.html#1844929 I've done some digging into this one and my notes are in the bug report. It mostly affects grenade jobs but is not entirely restricted to grenade jobs. It's also mostly on OVH and FortNebula nodes but not totally. From looking at the mysql logs in the grenade jobs, mysqld is (re)started three times. I think (1) for initial package install, (2) for stacking devstack on the old side, and (3) for stacking devstack on the new side. After the last restart, there are a lot of aborted connection messages in the msyql error log. It's around then that grenade is running post-upgrade smoke tests to create a server and the nova-scheduler times out communicating with the nova_cell1 database. I have a few patches up to grenade/devstack [1] to try some things and get more msyql logs but so far they aren't really helpful. We need someone with some more mysql debugging experience to help here, maybe zzzeek or mordred? 3. CirrOS guest SSH issues There are several (some might be duplicates): http://status.openstack.org/elastic-recheck/index.html#1848078 http://status.openstack.org/elastic-recheck/index.html#1808010 (most hits) http://status.openstack.org/elastic-recheck/index.html#1463631 http://status.openstack.org/elastic-recheck/index.html#1849857 http://status.openstack.org/elastic-recheck/index.html#1737039 http://status.openstack.org/elastic-recheck/index.html#1840355 http://status.openstack.org/elastic-recheck/index.html#1843610 A few notes here. a) We're still using CirrOS 0.4.0 since Stein: https://review.opendev.org/#/c/521825/ And that image was published nearly 2 years ago and there are no newer versions on the CirrOS download site so we can't try a newer image to see if that fixes things. b) Some of the issues above are related to running out of disk in the guest. I'm not sure what is causing that, but I have posted a devstack patch that is related: https://review.opendev.org/#/c/690991 tl;dr before Stein the tempest flavors we used had disk=0 so nova would fit the root disk to the size of the image. Since Stein the tempest flavors specify root disk size (1GiB for the CirrOS images). My patch pads an extra 1GiB to the root disk on the tempest flavors. One side effect of that is the volumes tempest creates will go from 1GiB to 2GiB which could be a problem if a lot of tempest volume tests run at the same time, though we do have a volume group size of 24GB in gate runs so I think we're OK for now. I'm not sure my patch would help, but it's an idea. As for the other key generation and dhcp lease failures, I don't know what to do about those. We need more eyes on these issues to generate some ideas or see if we're doing something wrong in our tests, e.g. generating too much data for the config drive? Not using config drive in some cases? Metadata API server is too slow (note we cache the metadata since [2])? Other ideas on injecting logs for debug? [1] https://review.opendev.org/#/q/topic:bug/1844929+status:open [2] https://review.opendev.org/#/q/I9082be077b59acd3a39910fa64e29147cb5c2dd7 -- Thanks, Matt

7 23

Is there a limit on the number of VMs returned by "openstack server list --all-projects" ?
by Albert Braden 20 Dec '19

20 Dec '19

2 4

[fuel][fuel-ccp] Retire Fuel and fuel-ccp repositories
by Andreas Jaeger 20 Dec '19

20 Dec '19

The fuel project has been removed from from OpenStack in 2017 and has no action for some time, the fuel-ccp repositories are dead as well. I've talked with Sergey from Mirantis and he confirmed now that those can be retired. I'll take care of retiring these repositories now and will retire all fuel repos in OpenStack namespace as well as the fuel-ccp repositories in the x namespace. I'll use topic retire-fuel for the changes, Andreas P.S. full list of repos: openstack/fuel-agent openstack/fuel-astute openstack/fuel-dev-tools openstack/fuel-devops openstack/fuel-docs openstack/fuel-library openstack/fuel-main openstack/fuel-menu openstack/fuel-mirror openstack/fuel-nailgun-agent openstack/fuel-nailgun-extension-cluster-upgrade openstack/fuel-nailgun-extension-converted-serializers openstack/fuel-nailgun-extension-iac openstack/fuel-octane openstack/fuel-ostf openstack/fuel-plugin-murano openstack/fuel-plugin-murano-tests openstack/fuel-plugins openstack/fuel-qa openstack/fuel-specs openstack/fuel-stats openstack/fuel-ui openstack/fuel-upgrade openstack/fuel-virtualbox openstack/fuel-web openstack/python-fuelclient x/fuel-ccp x/fuel-ccp-ceph x/fuel-ccp-ci-config x/fuel-ccp-cinder x/fuel-ccp-debian-base x/fuel-ccp-designate x/fuel-ccp-elasticsearch x/fuel-ccp-entrypoint x/fuel-ccp-etcd x/fuel-ccp-galera x/fuel-ccp-glance x/fuel-ccp-grafana x/fuel-ccp-heat x/fuel-ccp-horizon x/fuel-ccp-installer x/fuel-ccp-ironic x/fuel-ccp-keystone x/fuel-ccp-mariadb x/fuel-ccp-memcached x/fuel-ccp-murano x/fuel-ccp-neutron x/fuel-ccp-nginx x/fuel-ccp-nova x/fuel-ccp-openstack-base x/fuel-ccp-rabbitmq x/fuel-ccp-rally x/fuel-ccp-sahara x/fuel-ccp-searchlight x/fuel-ccp-specs x/fuel-ccp-stacklight x/fuel-ccp-tests x/fuel-ccp-zmq -- Andreas Jaeger aj(a)suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB

2 9

[kolla] broken images pushed
by Marcin Juszkiewicz 20 Dec '19

20 Dec '19

We have merged publisher for aarch64 images and it turned out that it was done wrong way. Today both x86-64 and aarch64 images got published at same time so "kolla/debian-source-*" images are broken: 08:55 (0s) hrw@puchatek:~$ docker run --rm -it kolla/debian-source-heat-engine:master bash standard_init_linux.go:211: exec user process caused "exec format error" This is result of aarch64 image overwriting x86-64 one. We reverted arm64 publisher and will work on it in January. Sorry for potential problem.

1 1

[Horizon][Designate][Tacker][Tobiko][OSH][Ironic] Flaky jobs and retries
by Clark Boylan 20 Dec '19

20 Dec '19

Hello, Zuulv3 has a feature where it will rerun jobs that it thinks failed due to infrastructure problems up to some configured limit. For us the limit is 3 job attempts. The circumstances where Zuul thinks it has infrastructure issues are when pre-run playbooks fail (because these are expected to be stable), and when Ansible returns with an exit code signifying network connectivity issues. Recently I updated Zuul to report the job attempt in job inventories and am using that to report this data into logstash and elasticsearch. Now I can use queries like: 'zuul_attempts:"3" AND filename:"job-output.txt" AND message:"Job console starting"' to see which jobs are flaky and reattempting. I'm writing this email to report on some of what I've found in the hopes that the problems we have can be fixed and hopefully result in better jobs across the board. Designate Horizon Dashboard The nodejs jobs for stable/rocky expect to run against nodejs4 but run on Bionic which has no nodejs 4 packaging. This causes the pre-run to fail as it cannot install the requested package, https://zuul.opendev.org/t/openstack/build/498ebb052e2b4a3393b0939820ee8927…. Fix in progress here, https://review.opendev.org/#/c/699241/1, to pin to Xenial. Thank you! Horizon projects and dashboards may want to double check they don't have similar problems with nodejs version and node mismatches. Tacker tacker-functional-devstack-multinode-python3 (and possibly other tacker devstack jobs) attempt to download an openwrt image during devstack runs and the server isn't responding. This fails in pre-run, http://zuul.openstack.org/build/27ff514c77724250968d60469923f613/log/contro…, and results in the job being retried. I am not aware of a fix for this, but you'll likely need to find a new image host. Let the infra team know if we can help. Tobiko tobiko-devstack-faults-centos-7 is failing because this job runs on CentOS 7 using python2 but Nova needs python3.6 now. This fails in pre-run, https://zuul.opendev.org/t/openstack/build/8071780e7ba748169b447f1d42e069fc…, and forces the job to be rerun. I'm not sure what the best fix is here. Maybe run your jobs on Ubuntu until CentOS 8 is supported by devstack? OpenStack Helm Ansible has decided that become_user is not valid on include_role tasks. https://opendev.org/openstack/openstack-helm-infra/src/branch/master/roles/… seems to be the issue. This causes Airship's deckhand functional docker jobs to fail in pre-run, https://zuul.opendev.org/t/openstack/build/7493038ee1744465b2387b44e067d029…, and the jobs are retried. It is possible this is fallout from Zuul's default ansible version being bumped to 2.8 from 2.7. This also seems to affect OSH's kubernetes keystone auth job, https://zuul.opendev.org/t/openstack/build/68d937b3e5e3449cb6fe2e6947bbf0db… Ironic The Ironic example is a bit different and runs into some fun Ansible behaviors. Ironic's IPA multinode jobs (possibly others too) are filling the root disk of the test nodes on some clouds, http://paste.openstack.org/show/787641/. Ansible uses /tmp to bootstrap its ssh connectivity and when /tmp is on / and / is full that results in Ansible returning a network failure error code. This then causes Zuul to rerun the job. Unfortunately because Ansible thinks networking is broken we have to do special things in a Zuul cleanup playbook to double check disk usage and that info is only available on the Zuul executors. But I've double checked for you and can confirm this is what is happening. The fix here is to use the ephemeral disk which is mounted on /opt and contains much more disk space. Another option would be to reduce the size of Ironic's fake baremetal images. In any case they are aware of this and we should expect a fix soon. There are likely more cases I haven't found yet, but I wanted to point out this Zuul behavior to people. The intent is that it handle exceptional cases and not paper over failures that happen often or even 100% of the time. We should do our best to fix these problems when they pop up. Thank you to everyone that has already stepped up to help fix some of these! Clark

2 2

[glance] no meeting on 26 December 2019
by Abhishek Kekane 20 Dec '19

20 Dec '19

As discussed during yesterday's meeting, we will not hold the glance weekly meeting on 26th December 2019. The next meeting will be on 2 January 2020. Wish you all a happy christmas and new year in advance. Thanks & Best Regards, Abhishek Kekane

1 0

[ironic] - automated cleaning
by fsbiz＠yahoo.com 20 Dec '19

20 Dec '19

We have enabled automatic cleaning and I have noticed that our UEFI enabled nodes most often end up in "clean failed" state. Basic troubleshooting reveals that PXE correctly boots IPXE, but the IPXE goes and boots the OS instead of downloading the deployed_kernel / ramdisk. Here's the timeline of events.Dec 18 12:19:39 sc-ironic04 Node 56e58642-12ac-4455-bc95-2a328198f845 moved to provision state "cleaning"Dec 18 12:20:32 sc-ironic04 Successfully set node 56e58642-12ac-4455-bc95-2a328198f845 power state to power onDec 18 12:21:14 DHCPACK(ns-601d0738-39) 10.33.23.7 6c:b3:11:4f:8b:18 (PXE Boot gets IP: 10.33.23.7)Dec 18 12:21:15 sc-ironic04 in.tftpd[367896]: Client 10.33.23.7 finished ipxe_x86_64.efi (TFTP of IPXE is complete)Dec 18 12:21:23 DHCPACK(ns-261f35c5-7e) 10.33.23.7 6c:b3:11:4f:8b:18 host-10-33-23-7 (IPXE acquires IP: 10.33.23.7)NO HTTP request for deploy_kernel Instead:Dec 18 12:22:40 sc-control04 dnsmasq-dhcp[3508449]: 3990148443 DHCPREQUEST(ns-261f35c5-7e) 10.33.22.188 6c:b3:11:4f:8b:18Dec 18 12:22:40 sc-control04 dnsmasq-dhcp[3508449]: 3990148443 DHCPNAK(ns-261f35c5-7e) 10.33.22.188 6c:b3:11:4f:8b:18 wrong address When the OS was running it had IP address of 10.33.22.188. The OS came up and tries to renew its lease for 10.33.22.188 which was NAKed by the DHCP server. The OS then did a full DHCP (DISCOVER, etc.), got a valid IP address, did a cloud-init, etc. Is this a known issue that has been fixed ? Any pointers would be appreciated. thanks,Farad.

1 0