- openstack-discuss - lists.openstack.org

[Horizon][Designate][Tacker][Tobiko][OSH][Ironic] Flaky jobs and retries
by Clark Boylan 20 Dec '19

20 Dec '19

Hello, Zuulv3 has a feature where it will rerun jobs that it thinks failed due to infrastructure problems up to some configured limit. For us the limit is 3 job attempts. The circumstances where Zuul thinks it has infrastructure issues are when pre-run playbooks fail (because these are expected to be stable), and when Ansible returns with an exit code signifying network connectivity issues. Recently I updated Zuul to report the job attempt in job inventories and am using that to report this data into logstash and elasticsearch. Now I can use queries like: 'zuul_attempts:"3" AND filename:"job-output.txt" AND message:"Job console starting"' to see which jobs are flaky and reattempting. I'm writing this email to report on some of what I've found in the hopes that the problems we have can be fixed and hopefully result in better jobs across the board. Designate Horizon Dashboard The nodejs jobs for stable/rocky expect to run against nodejs4 but run on Bionic which has no nodejs 4 packaging. This causes the pre-run to fail as it cannot install the requested package, https://zuul.opendev.org/t/openstack/build/498ebb052e2b4a3393b0939820ee8927…. Fix in progress here, https://review.opendev.org/#/c/699241/1, to pin to Xenial. Thank you! Horizon projects and dashboards may want to double check they don't have similar problems with nodejs version and node mismatches. Tacker tacker-functional-devstack-multinode-python3 (and possibly other tacker devstack jobs) attempt to download an openwrt image during devstack runs and the server isn't responding. This fails in pre-run, http://zuul.openstack.org/build/27ff514c77724250968d60469923f613/log/contro…, and results in the job being retried. I am not aware of a fix for this, but you'll likely need to find a new image host. Let the infra team know if we can help. Tobiko tobiko-devstack-faults-centos-7 is failing because this job runs on CentOS 7 using python2 but Nova needs python3.6 now. This fails in pre-run, https://zuul.opendev.org/t/openstack/build/8071780e7ba748169b447f1d42e069fc…, and forces the job to be rerun. I'm not sure what the best fix is here. Maybe run your jobs on Ubuntu until CentOS 8 is supported by devstack? OpenStack Helm Ansible has decided that become_user is not valid on include_role tasks. https://opendev.org/openstack/openstack-helm-infra/src/branch/master/roles/… seems to be the issue. This causes Airship's deckhand functional docker jobs to fail in pre-run, https://zuul.opendev.org/t/openstack/build/7493038ee1744465b2387b44e067d029…, and the jobs are retried. It is possible this is fallout from Zuul's default ansible version being bumped to 2.8 from 2.7. This also seems to affect OSH's kubernetes keystone auth job, https://zuul.opendev.org/t/openstack/build/68d937b3e5e3449cb6fe2e6947bbf0db… Ironic The Ironic example is a bit different and runs into some fun Ansible behaviors. Ironic's IPA multinode jobs (possibly others too) are filling the root disk of the test nodes on some clouds, http://paste.openstack.org/show/787641/. Ansible uses /tmp to bootstrap its ssh connectivity and when /tmp is on / and / is full that results in Ansible returning a network failure error code. This then causes Zuul to rerun the job. Unfortunately because Ansible thinks networking is broken we have to do special things in a Zuul cleanup playbook to double check disk usage and that info is only available on the Zuul executors. But I've double checked for you and can confirm this is what is happening. The fix here is to use the ephemeral disk which is mounted on /opt and contains much more disk space. Another option would be to reduce the size of Ironic's fake baremetal images. In any case they are aware of this and we should expect a fix soon. There are likely more cases I haven't found yet, but I wanted to point out this Zuul behavior to people. The intent is that it handle exceptional cases and not paper over failures that happen often or even 100% of the time. We should do our best to fix these problems when they pop up. Thank you to everyone that has already stepped up to help fix some of these! Clark

2 2

[glance] no meeting on 26 December 2019
by Abhishek Kekane 20 Dec '19

20 Dec '19

As discussed during yesterday's meeting, we will not hold the glance weekly meeting on 26th December 2019. The next meeting will be on 2 January 2020. Wish you all a happy christmas and new year in advance. Thanks & Best Regards, Abhishek Kekane

1 0

[ironic] - automated cleaning
by fsbiz＠yahoo.com 20 Dec '19

20 Dec '19

We have enabled automatic cleaning and I have noticed that our UEFI enabled nodes most often end up in "clean failed" state. Basic troubleshooting reveals that PXE correctly boots IPXE, but the IPXE goes and boots the OS instead of downloading the deployed_kernel / ramdisk. Here's the timeline of events.Dec 18 12:19:39 sc-ironic04 Node 56e58642-12ac-4455-bc95-2a328198f845 moved to provision state "cleaning"Dec 18 12:20:32 sc-ironic04 Successfully set node 56e58642-12ac-4455-bc95-2a328198f845 power state to power onDec 18 12:21:14 DHCPACK(ns-601d0738-39) 10.33.23.7 6c:b3:11:4f:8b:18 (PXE Boot gets IP: 10.33.23.7)Dec 18 12:21:15 sc-ironic04 in.tftpd[367896]: Client 10.33.23.7 finished ipxe_x86_64.efi (TFTP of IPXE is complete)Dec 18 12:21:23 DHCPACK(ns-261f35c5-7e) 10.33.23.7 6c:b3:11:4f:8b:18 host-10-33-23-7 (IPXE acquires IP: 10.33.23.7)NO HTTP request for deploy_kernel Instead:Dec 18 12:22:40 sc-control04 dnsmasq-dhcp[3508449]: 3990148443 DHCPREQUEST(ns-261f35c5-7e) 10.33.22.188 6c:b3:11:4f:8b:18Dec 18 12:22:40 sc-control04 dnsmasq-dhcp[3508449]: 3990148443 DHCPNAK(ns-261f35c5-7e) 10.33.22.188 6c:b3:11:4f:8b:18 wrong address When the OS was running it had IP address of 10.33.22.188. The OS came up and tries to renew its lease for 10.33.22.188 which was NAKed by the DHCP server. The OS then did a full DHCP (DISCOVER, etc.), got a valid IP address, did a cloud-init, etc. Is this a known issue that has been fixed ? Any pointers would be appreciated. thanks,Farad.

1 0

[magnum] Next weekly team meeting will be on 22 Jan 2020
by Feilong Wang 20 Dec '19

20 Dec '19

Hi team, We just had a good year and we deserve some rest. We will skip some meetings and the next one will be on 22 Jan 2020. Hope you all will have a merry Christmas and happy New Year. -- Cheers & Best regards, Feilong Wang (王飞龙) Head of R&D Catalyst Cloud - Cloud Native New Zealand -------------------------------------------------------------------------- Tel: +64-48032246 Email: flwang(a)catalyst.net.nz Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------

1 0

[qa][nova][cinder] ceph job has been busted on stable/pike for awhile, fix or delete?
by Matt Riedemann 20 Dec '19

20 Dec '19

The ceph job on stable/pike has been broken for awhile [1]. The fix to make devstack-plugin-ceph (which is branchless) work on stable/pike depends on this devstack change on stable/pike to introduce the TARGET_BRANCH variable [2]. Question is, can we move [2] forward? If not, I make a motion that we drop ceph jobs from stable/pike (and likely stable/ocata if we are still running them there) because they won't work. The ceph job is non-voting anyway so that's why it goes unnoticed for so long when it's broken. If we aren't going to fix this stuff we shouldn't be wasting CI resources running broken jobs. Did I mention that you should all get off my lawn? :) [1] https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1835627 [2] https://review.opendev.org/#/c/684756/ -- Thanks, Matt

5 7

[API SIG] holiday office hours
by Michael McCune 20 Dec '19

20 Dec '19

the API SIG will be closing it's office hours until the new year. have a peaceful and joyous holiday season and new years =) peace o/

1 0

[blazar] IRC meetings cancelled during the holidays
by Pierre Riteau 19 Dec '19

19 Dec '19

Hello, The Blazar IRC meetings of December 24, December 31, and January 2 are cancelled. The next meeting will be on Tuesday, January 7. Have a great holiday, Pierre

1 0

[manila] no weekly meeting 26 December 2019
by Tom Barron 19 Dec '19

19 Dec '19

Just noting that there will be no weekly Manila meeting 26 December. We'll next meet at the scheduled time, 1500 UTC, on freenode #openstack-meeting-alt, on 2 January 2020. [1] See you in the New Year, or in #openstack-manila in the mean time! -- Tom Barron [1] https://wiki.openstack.org/wiki/Manila/Meetings

1 0

CfP VHPC20: HPC Containers-Kubernetes
by VHPC 20 19 Dec '19

19 Dec '19

==================================================================== CALL FOR PAPERSa 15th Workshop on Virtualization in High-Performance Cloud Computing (VHPC 20) held in conjunction with the International Supercomputing Conference - High Performance, June 21-25, 2020, Frankfurt, Germany. (Springer LNCS Proceedings) ==================================================================== Date: June 25, 2020 Workshop URL: vhpc[dot]org Abstract registration Deadline: Jan 31st, 2020 Paper Submission Deadline: Apr 05th, 2020 Springer LNCS Call for Papers Containers and virtualization technologies constitute key enabling factors for flexible resource management in modern data centers, and particularly in cloud environments. Cloud providers need to manage complex infrastructures in a seamless fashion to support the highly dynamic and heterogeneous workloads and hosted applications customers deploy. Similarly, HPC environments have been increasingly adopting techniques that enable flexible management of vast computing and networking resources, close to marginal provisioning cost, which is unprecedented in the history of scientific and commercial computing. Most recently, Function as a Service (Faas) and Serverless computing, utilizing lightweight VMs-containers widens the spectrum of applications that can be deployed in a cloud environment, especially in an HPC context. Here, HPC-provided services can be become accessible to distributed workloads outside of large cluster environments. Various virtualization-containerization technologies contribute to the overall picture in different ways: machine virtualization, with its capability to enable consolidation of multiple underutilized servers with heterogeneous software and operating systems (OSes), and its capability to live-migrate a fully operating virtual machine (VM) with a very short downtime, enables novel and dynamic ways to manage physical servers; OS-level virtualization (i.e., containerization), with its capability to isolate multiple user-space environments and to allow for their coexistence within the same OS kernel, promises to provide many of the advantages of machine virtualization with high levels of responsiveness and performance; lastly, unikernels provide for many virtualization benefits with a minimized OS/library surface. I/O Virtualization in turn allows physical network interfaces to take traffic from multiple VMs or containers; network virtualization, with its capability to create logical network overlays that are independent of the underlying physical topology is furthermore enabling virtualization of HPC infrastructures. Publication Accepted papers will be published in a Springer LNCS proceedings volume. Topics of Interest The VHPC program committee solicits original, high-quality submissions related to virtualization across the entire software stack with a special focus on the intersection of HPC, containers-virtualization and the cloud. Major Topics: - HPC workload orchestration (Kubernetes) - Kubernetes HPC batch - HPC Container Environments Landscape - HW Heterogeneity - Container ecosystem (Docker alternatives) - Networking - Lightweight Virtualization - Unikernels / LibOS - State-of-the-art processor virtualization (RISC-V, EPI) - Containerizing HPC Stacks/Apps/Codes: Climate model containers each major topic encompassing design/architecture, management, performance management, modeling and configuration/tooling. Specifically, we invite papers that deal with the following topics: - HPC orchestration (Kubernetes) - Virtualizing Kubernetes for HPC - Deployment paradigms - Multitenancy - Serverless - Declerative data center integration - Network provisioning - Storage - OCI i.a. images - Isolation/security - HW Accelerators, including GPUs, FPGAs, AI, and others - State-of-practice/art, including transition to cloud - Frameworks, system software - Programming models, runtime systems, and APIs to facilitate cloud adoption - Edge use-cases - Application adaptation, success stories - Kubernetes Batch - Scheduling, job management - Execution paradigm - workflow - Data management - Deployment paradigm - Multi-cluster/scalability - Performance improvement - Workflow / execution paradigm - Podman: end-to-end Docker alternative container environment & use-cases - Creating, Running containers as non-root (rootless) - Running rootless containers with MPI - Container live migration - Running containers in restricted environments without setuid - Networking - Software defined networks and network virtualization - New virtualization NICs/Nitro alike ASICs for the data center? - Kubernetes SDN policy (Calico i.a.) - Kubernetes network provisioning (Flannel i.a.) - Lightweight Virtualization - Micro VMMs (Rust-VMM, Firecracker, solo5) - Xen - Nitro hypervisor (KVM) - RVirt - Cloud Hypervisor - Unikernels / LibOS - HPC Storage in Virtualization - HPC container storage - Cloud-native storage - Hypervisors in storage virtualization - Processor Virtualization - RISC-V hypervisor extensions - RISC-V Hypervisor ports - EPI - Composable HPC microservices - Containerizing Scientific Codes - Building - Deploying - Securing - Storage - Monitoring - Use case for containerizing HPC codes: Climate model containers for portability, reproducibility, traceability, immutability, provenance, data & software preservation The Workshop on Virtualization in High-Performance Cloud Computing (VHPC) aims to bring together researchers and industrial practitioners facing the challenges posed by virtualization in order to foster discussion, collaboration, mutual exchange of knowledge and experience, enabling research to ultimately provide novel solutions for virtualized computing systems of tomorrow. The workshop will be one day in length, composed of 20 min paper presentations, each followed by 10 min discussion sections, plus lightning talks that are limited to 5 minutes. Presentations may be accompanied by interactive demonstrations. Important Dates Jan 31st, 2020 - Abstract Apr 5th, 2020 - Paper submission deadline (Springer LNCS) Apr 26th, 2020 - Acceptance notification June 25th, 2020 - Workshop Day July 10th, 2020 - Camera-ready version due Chair Michael Alexander (chair), BOKU, Vienna, Austria Anastassios Nanos (co-chair), Sunlight.io, UK Program committee Stergios Anastasiadis, University of Ioannina, Greece Paolo Bonzini, Redhat, Italy Jakob Blomer, CERN, Europe Eduardo César, Universidad Autonoma de Barcelona, Spain Taylor Childers, Argonne National Laboratory, USA Stephen Crago, USC ISI, USA Tommaso Cucinotta, St. Anna School of Advanced Studies, Italy François Diakhaté CEA DAM Ile de France, France Kyle Hale, Northwestern University, USA Brian Kocoloski, Washington University, USA John Lange, University of Pittsburgh, USA Giuseppe Lettieri, University of Pisa, Italy Klaus Ma, Huawei, China Alberto Madonna, Swiss National Supercomputing Center, Switzerland Nikos Parlavantzas, IRISA, France Anup Patel, Western Digital, USA Kevin Pedretti, Sandia National Laboratories, USA Amer Qouneh, Western New England University, USA Carlos Reaño, Queen’s University Belfast, UK Adrian Reber, Redhat, Germany Riccardo Rocha, CERN, Europe Borja Sotomayor, University of Chicago, USA Jonathan Sparks, Cray, USA Kurt Tutschku, Blekinge Institute of Technology, Sweden John Walters, USC ISI, USA Yasuhiro Watashiba, Osaka University, Japan Chao-Tung Yang, Tunghai University, Taiwan Paper Submission-Publication Papers submitted to the workshop will be reviewed by at least two members of the program committee and external reviewers. Submissions should include abstract, keywords, the e-mail address of the corresponding author, and must not exceed 10 pages, including tables and figures at a main font size no smaller than 11 point. Submission of a paper should be regarded as a commitment that, should the paper be accepted, at least one of the authors will register and attend the conference to present the work. Accepted papers will be published in a Springer LNCS volume. The format must be according to the Springer LNCS Style. Initial submissions are in PDF; authors of accepted papers will be requested to provide source files. Abstract, Paper Submission Link: edas[dot]info/newPaper.php?c=26973 Lightning Talks Lightning Talks are non-paper track, synoptical in nature and are strictly limited to 5 minutes. They can be used to gain early feedback on ongoing research, for demonstrations, to present research results, early research ideas, perspectives and positions of interest to the community. Submit abstract via the main submission link. General Information The workshop is one day in length and will be held in conjunction with the International Supercomputing Conference - High Performance (ISC) 2019, June 21-25, Frankfurt, Germany.

1 0

[neutron] PTL on PTO
by Slawek Kaplonski 19 Dec '19

19 Dec '19

Hi, I will be on PTO from Friday 20.12.2019 up to 6.01.2020. I will probably not be available on IRC for most of the time but I will be checking emails from time to time. So if You have anything important, please email me. — Slawek Kaplonski Senior software engineer Red Hat

1 0

[neutron]CI meeting cancelled
by Slawek Kaplonski 19 Dec '19

19 Dec '19

Hi, Our Neutron CI meetings which should be on 24.12.2019 and 31.12.2019 will be cancelled. See You all in new year :) Have a great holiday time. — Slawek Kaplonski Senior software engineer Red Hat

1 0

[neutron] Team meeting cancelled
by Slawek Kaplonski 19 Dec '19

19 Dec '19

Hi, Our Neutron team meetings which should be on 23.12.2019 and 31.12.2019 will be cancelled. See You all in new year :) Have a great holiday time. — Slawek Kaplonski Senior software engineer Red Hat

1 0

[keystone] No meetings over the holidays
by Colleen Murphy 19 Dec '19

19 Dec '19

As discussed at yesterday's meeting, we'll cancel keystone team meetings for the next two weeks for the winter holidays. The next keystone meeting will be Tuesday, January 7 (17:00 UTC, #openstack-meeting-alt). Colleen

1 0

[resource-management-sig] Status of the "Resource Management" SIG
by Thierry Carrez 19 Dec '19

19 Dec '19

Hi, I was wondering about the status of the Resource Management SIG... It's been "forming" according to https://wiki.openstack.org/wiki/Res_Mgmt_SIG since January 2018... And I could'nt find a reference or log to any meeting after that. Does anyone have updated status on this one? Should it be removed from the list of active SIGs at https://governance.openstack.org/sigs/ ? -- Thierry Carrez (ttx)

5 6

[bifrost][ironic][kayobe][kolla][requirements] Kayobe CI breakage following ironic-lib release
by Pierre Riteau 19 Dec '19

19 Dec '19

Hello, I have been looking into CI failures that started impacting Kayobe today on the master branch, just as we're planning to branch stable/train this week (Kayobe is cycle trailing). Our seed jobs are failing because ironic doesn't start correctly [1]. The traceback shows that it is using code from the latest ironic-lib (3.0.0) which was released just last week [2] and dropped Python 2 compatibility. We're using Kolla Train images [3], but the latest kolla/centos-source-bifrost-deploy:train includes Bifrost 7.0.0 which fetches all its dependencies using Git master branches [4]. This only got changed later on stable/train [5], but there is no newer 7.x release. One of these dependencies is the requirements repo, which has just bumped the upper constraint for ironic-lib to 3.0.0 [6]. I believe this is what caused our jobs to start failing today. So I think we need: - a 7.0.1 release of Bifrost - an update of Kolla to use this release instead of 7.0.0 Would the ironic and kolla project teams be willing to help with this to fix our gate? Thanks a lot. [1] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackc… [2] https://review.opendev.org/#/c/698500/ [3] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackc… [4] https://opendev.org/openstack/bifrost/src/tag/7.0.0/playbooks/roles/bifrost… [5] https://review.opendev.org/#/q/I3005fc59d6c5e1f5000874342f956977d9929907 [6] https://review.opendev.org/#/c/698981/

2 4

[ironic] ibmc driver
by Julia Kreger 18 Dec '19

18 Dec '19

Greetings fellow ironicans and stackers! I'd like some thoughts on the `ibmc` driver[1] that was added early this year. Specifically it merged into ironic with an operating Third-Party CI, however soon after a series of unfortunate events occurred where the Third-Party CI stopped reporting sometime in early May, and shortly their after restrictions were put into place[2]. With that being said, I feel like we should go ahead and remove this driver since it does not seem to be maintained nor tested. -Julia [1]: https://review.opendev.org/#/c/639288/ [2]: https://www.federalregister.gov/documents/2019/05/21/2019-10616/addition-of…

3 4

[fuel] fuel-plugins, x/network-checker, x/fuel-noop-fixtures retirement
by Andreas Jaeger 18 Dec '19

18 Dec '19

Since we have now the fuel repositories retired, I propose to retire the fuel repos in the x/ namespace as well: all the plugins, network-checker and fuel-noop-fixuters There was no single merge to them since October 2017 (with exception of Ian's force-merge of URL replacements), see https://review.opendev.org/#/q/(projects:x/fuel-plugin+OR+project:x/network… I will push changes to repositories and would appreciate if teams merge them and abandon all open reviews. If anybody wants to keep the repos, please speak up here and/or -1 my changes. I'll use topic retire-x-fuel, https://review.opendev.org/#/q/topic:retire-x-fuel I'll just propose changes for now, we'll force-merge them in January 2020, Andreas -- Andreas Jaeger aj(a)suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB

1 0

[cinder] next meeting 8 January 2020
by Brian Rosmaita 18 Dec '19

18 Dec '19

As discussed during today's meeting, we will *not* hold the Cinder weekly meeting over the next 2 weeks. The next meeting will be on 8 January 2020. https://etherpad.openstack.org/p/cinder-ussuri-meetings cheers, brian

1 0

[tripleo] Gate is unstable, do not recheck/approve anything now
by Emilien Macchi 18 Dec '19

18 Dec '19

We're encountering a bug in TripleO, see https://bugs.launchpad.net/tripleo/+bug/1856864 In the meantime please do not recheck or approve a patch, the gate is already fully loaded; we'll purge it, fix the bug and recheck what needs to be done. You can follow the progress of that work on our IRC channel #tripleo. Thanks for your patience. -- Emilien Macchi

1 0

[neutron] Cancle neutron L3 meeting
by LIU Yulong 18 Dec '19

18 Dec '19

Hi all, Due to some local reasons, I'm not be able to host the L3 meeting today. So let's cancel it. And also because contributors from most countries will have the Christmas and new year's holidays in the next two weeks, let's also cancel the following two L3 meetings. So the next L3 meeting will be raised on 8th Jan, 2020. OK, see you guys then.  And happy new year! Regards, LIU Yulong

1 0

[kolla][horizon] Get 503 Service unavailable after set RETRIEVE_PASSWORD.
by Eddie Yen 18 Dec '19

18 Dec '19

Hi, Below is the information about this environment. Openstack release: Rocky Kolla container source type: binary OS distro: Ubuntu 18.04 We tried to set OPENSTACK_ENABLE_PASSWORD_RETRIEVE to True in local_settings, but the dashboard got "503 Service Unavailable" after restart the Horizon container. And there's no error log or something in horizon.log Is there anyway to see what's going on? - Eddie

2 2

[openstack-dev][kuryr] Unable to connect to kube-apiserver
by Veera.reddy B 18 Dec '19

18 Dec '19

Hi , I install Kubernetes using "kubeadm init". Followed below link to install "kuryr-kubernetes". https://docs.openstack.org/kuryr-kubernetes/latest/installation/manual.html… Kube api configuration : /etc/kubernetes/manifests/kube-apiserver.yaml http://paste.openstack.org/show/787704/ kuryr config file : /etc/kuryr/kuryr.conf http://paste.openstack.org/show/787705/ Error while starting kuryr : #> kuryr-k8s-controller --config-file /etc/kuryr/kuryr.conf http://paste.openstack.org/show/787706/ Let me what I am missing in "/etc/kuryr/kuryr.conf" Thanks, Veera.

2 1

[openstack-dev][kuryr] Unable to connect to kube-apiserver
by VeeraReddy 18 Dec '19

18 Dec '19

Hi , I install Kubernetes using “kubeadm init”. Followed below link to install “kuryr-kubernetes”. https://docs.openstack.org/kuryr-kubernetes/latest/installation/manual.html… Kube api configuration : /etc/kubernetes/manifests/kube-apiserver.yaml http://paste.openstack.org/show/787704/ kuryr config file : /etc/kuryr/kuryr.conf http://paste.openstack.org/show/787705/ Error while starting kuryr : #> kuryr-k8s-controller--config-file /etc/kuryr/kuryr.conf http://paste.openstack.org/show/787706/ Let me what I am missing in “/etc/kuryr/kuryr.conf” Thanks, Veera.

1 0

[scientific-sig] No IRC meeting today
by Stig Telfer 18 Dec '19

18 Dec '19

Hi All - Apologies, in the rush to end of year we have no agenda items for today. Given the season, I suggest we reconvene with a fresh start on Tuesday 7th January 2020. Cheers, Stig

1 0

[Watcher]IRC meeting at 8:00 UTC today
by licanwei 18 Dec '19

18 Dec '19

1 0