May 2019 - openstack-discuss - lists.openstack.org

[manila] PTG Summary
by Victoria Martínez de la Cruz 14 May '19

14 May '19

Hi all, Just reaching you out with a brief PTG summary for the Manila team. Thanks to everyone who attended and participated in-person and remote! Here's the PTG notes etherpad: https://etherpad.openstack.org/p/manila-ptg-train *High points* - We had a highly productive PTG, albeit shorter than the previous times we've met as a group. It reminded us of the old Design Summit times, although we weren't doing smaller time slots during the Summit week, but rather had a separate time to catch up. - We had ample time to do cross-project sync up; however, there were unavoidable overlaps. - It was also great to have many engineers travel to both the summit and the PTG. The following is a summary of notes organized with action items and owner/s. *Cross project goals* - Summary: - Identified current cross project goals, being them PDF rendering and IPv6 support - For the first case, we rely on the current documentation we have and on the docs team to guide us on the changes we need to do to have PDF rendering support - For the latter, we have been supporting IPv6 on manila for a while now and we have CI enabled to exercise this configuration. - Action: - Work with asettle to understand what it's need to be done to comply with the PDF rendering goal - Create an IPv6 only job in manila - many of our jobs run with 4+6, where we're currently doing IPv6 data path testing. - Owner: Not yet defined *Auto-snapshotting of manila shares* - Summary: - CERN had a use case that required cloud administrator driven automatic snapshots at pre-configured intervals, with pre-configured retention policies. - The team discussed ways the automation can occur with the existing API. - There are some backend specific extra-specs that can allow offloading the responsibility of periodic snapshots, and pruning of snapshots per retention policy. There is a disadvantage here that these snapshots are invisible to manila; and do not count against the project quotas - They also require knowing backend special sauce. - Snapshotting of multiple shares can be achieved with Share Groups - To make this occur without end-user actions, one would need to build tooling with these APIs - Maybe Mistral can be harnessed to do workflow scheduling and automation per business logic. - Action: CERN will explore these options and revert with their thoughts - Owner/s: CERN/Jose Castro Leon/josecastroleon *Manila in k8s* - Summary: - Robert Vasek/gman0 has an initial implementation of the CSI driver on the cloud-provider-openstack repo. We're seeking reviewers to merge it - https://github.com/kubernetes/cloud-provider-openstack/pull/536 - Next set of TODOs: - Robert's application to GSOC is pending, if he gets sponsorship, he might lead some of the following efforts: - e2e testing in the OpenLab CI for the manila-csi-provisioner - Testing with an NFS backend - AZ awareness / Topology based scheduling support - Snapshot support - Volume expansion - NoAuth mode for using the provisioner in single-tenant deployments - Manila differentiates abilities to take snapshots of shares, cloning these snapshots into new shares, mounting snapshots (with access control) and reverting shares to snapshots in place. CephFS supports snapshots, however, does not support rapid cloning - this feature gap exists for OpenStack use case as well. - Plans to build snapshot cloning capability into Manila and generically within the CSI provisioner were discussed. We'll evolve this design in the coming months. - Action Items: - Continue on plan to build and support manila-csi-provisioner (gman0, Tomas Smatena/tsmatena, Goutham Ravi/gouthamr, Tom Barron/tbarron, Victoria Martinez de la Cruz/vkmc, manila community) - Gather user feedback for the usecase for CEPH snapshots (josecastroleon and Xing Yang/xyang - Ask CERN and Huawei if they care about a slower "create-share-from-snapshot" over not having the ability at all. - Owners: as identified above ^ *OpenStack Client and OpenStack SDK* - Summary: - Manila lags in supporting OSC. Work began late in the Stein cycle to bridge this gap: - Specification: https://review.opendev.org/#/c/644218/ - Initial Code: https://review.opendev.org/#/c/642222/ - We have an Outreachy Intern starting with the manila team (Soledad Kuczala/s0ru) - Sofia Enriquez/enriquetaso and vkmc will be mentoring the intern - CERN may offer cycles to cover some of the development work - Amit Oren/amito submitted changes to add manila support to the openstacksdk: - *https://review.opendev.org/#/c/638782/* <https://review.opendev.org/#/c/638782/> - Team agreed to use the existing manilaclient SDK (in python-manilaclient) for the OSC instead of using the openstacksdk right away until we achieve feature parity in the openstacksdk project - Action Items: - Start Working Group to plan/coordinate OSC implementation - Identify minimum set of commands to implement and extended set to achieve parity with existing client - Allow support for using "openstack share" and "manila" to invoke manila CLI commands - the latter is essential to support manila running standalone. - Deprecate the existing manila clients in favor of the newer OpenStack CLI (and corresponding manila bash shell equivalent) - Review the openstacksdk change and identify feature gaps - Owner/s: s0ru, enriquetaso, vkmc, amito, gouthamr, josecastroleon *SemVer in manila releases* - Summary: This was a recap of how we are versioning the manila deliverables as we release them - "manila" gets a major version bump every release - Client releases indicate backwards compatibility by not bumping up the major release version. Minor version (and micro-version) bumps are done per policy based off of what changes are being released - The general guideline to follow is that of pbr versioning notes here: https://docs.openstack.org/pbr/latest/user/semver.html - Defer to the judgement of the OpenStack release team when in doubt - Action items: None - Owner/s: None *Create share from snapshots in another pool or back end* - Summary: - This specification has new owners who explained how they plan to implement this feature - The main concerns for the proposed spec ( https://review.opendev.org/#/c/609537/) were around the user experience and administrator control of the scheduling - Existing scheduling capabilities will be used to determine when a snapshot has to be cloned into a new pool rather than its existing pool. - Users can influence the scheduler's decision by specifying the availability zone or a different share type - Implementation will be two-phased - a driver optimized approach followed by a generic approach - Action Items: - Re-target specification to Train, assign to new owners - Implement driver-driven snapshot cloning to different pool - Owner/s: Douglas Viroel/dviroel, Lucio Seki/lseki, Erlon Cruz/erlon *Security Service password* - Summary: - Manila stores the security service password as plaintext in its database. It also allows all users within the tenant to retrieve the password via the security service API. - https://bugs.launchpad.net/manila/+bug/1817316 - We discussed adding barbican and castellan support to manila and making it a hard dependency by including their clients into the requirements.txt - Appropriate key management software will be a soft dependency, and there will be a recommendation about which key management backend can be used - a soft dependency to manila. - Action Items: - Disallow retrieving password from the security services API - Investigate use of Barbican/Castellan to store security services API password - change representation in the database - Owner/s: Not yet identified *Edge Requirements and active/active* - Summary: - Active-active - Manila share manager service running active/active - this has been a request from a long time, from traditional data center workloads as well as for newer/Edge workloads - Ganesha active/active - this allows datapath high availability for the CephFS-NFS driver - Edge - The goal of edge is to distribute services better, with latency considerations. We rely in deployment tools such as TripleO to do this work. This implementation has started already in some other projects (e.g. Cinder). TripleO supports and edge topology already. It's mainly hyperconverged OpenStack at the edge. - Initial users are telco customers running NFVs, 5G - Other services have been configured to be deployed on the edge - Cinder started on this effort last cycle - Cinder - qualify at least one driver to work in an active-active volume service - Is tooz supported for cinder-volume at the edge? - What tooz backend is being deployed? -> etcd deployed at each edge - Action items: - Introduce a multi-node job in the manila repository to test active-active deployment in the gate - Audit the usage of oslo_concurrency based file locks in the share manager service, replace these with tooz based locks - Design a leader election based workflow for asynchronous/time based tasks occuring within the share manager - Owner/s: gouthamr *OpenStack technical vision* - Summary: - This was a discussion of the TC OpenStack Vision Document: https://governance.openstack.org/tc/reference/technical-vision.html - Tom submitted a proposal that is for review in https://review.openstack.org/#/c/636770/. Based on this we exchanged some ideas regarding the accuracy of this document. - Action items: Reviewers are needed to check this proposal. - Owner/s: tbarron *API Improvements: Pagination and Filtering* - Summary: In the last couple of cycles several bugs were filed wrt pagination and filtering for our API. Precisely, - Can't filter/list resources that have no key=value metadata/extra-specs (https://bugs.launchpad.net/manila/+bug/1782847) - Pagination does not speed up list queries ( https://bugs.launchpad.net/manila/+bug/1795463) - next share_links contains not supported marker field ( https://bugs.launchpad.net/manila/+bug/1819167) - Action items: Look for volunteers wishing to work on the reported bugs. Bug/1795463 is being followed up by carloss. - Owner/s: Each bug has its asignee *CI Improvements: Rationalizing our jobs* - Summary: This discussion focused on the current status of our CI. It's fairly complex and we noticed that we need to simplify it to focus our resources better. Main highlights are: - We test 8 FOSS/first party drivers (Generic DHSS=True, Generic DHSS=False, Container, LVM, ZFSOnLinux, CEPHFS-Native, CEPHFS-NFS, ~~~Dummy~~~) - We test two databases - mysql, postgresql - We run API and scenario tests (together in some cases, individually as "manila-tempest-dsvm-scenario" (Generic driver DHSS=True) - We run python2 and python3 jobs - We run same jobs in manila-tempest-plugin and manila repos - Action items: - Generic driver with DHSS=False will be removed; - the job for Generic driver with DHSS=True can be combined with dsvm-scenario. - Unify tempest api and scenario tests for all drivers, even if that means we must bump timeouts. - Owner/s: gouthamr *Generic Driver improvements* - Summary: - There are lot of issues (even for usage at the gate) wrt the Generic Driver and nobody has enough bandwitch to work on all of them. Some of the known issues are: - It has a single point of failure, since there is no HA for the share servers created by manila - Attach / re-attach issues - timeouts - We need to figure out is there are people using the generic driver, understand better if we need to keep it or if it can be substituted for a different driver. Ben Swartzlander had a "NextGen" generic driver that is stuck in review because there's no migration path to move shares between the two different implementations of the generic driver: - https://review.opendev.org/#/c/511038/ - There are some production users of the Generic Driver that seem to have solved the SPOF (Single point of failure) issues in private forks of manila. If they are interested in sharing the love upstream, the manila community will be very welcoming. - Action items: - Continue to look for alternatives to the generic driver for CI. - Discuss in following upstream Manila meetings. - Owner/s: None *Bugs* - Summary: - Force flag on the snapshot creation API ( https://bugs.launchpad.net/manila/+bug/1811336) Considered to be a doc bug. Need to clarify this in the bug report. Marked as a "low-hanging-fruit" for new volunteers. - Limit share size (https://bugs.launchpad.net/manila/+bug/1811943) The team is interested in this one, we replied to the reporter during the PTG. We need to keep the dicussion going and look for a volunteer to work on this. - Incompatible protocol/access types in the API ( https://bugs.launchpad.net/manila/+bug/1637542) - Action items: None in particular, triaged during the PTG - Owner/s: None Cheers, Victoria

1 0

[nova] Bug warning: function wrapped in db retry which modifies its arguments
by Matthew Booth 14 May '19

14 May '19

I'm sharing this because I have a suspicion it's a class of bug rather than just this one, but I haven't gone looking. The pattern is: @wrap_db_retry def instance_update_and_get_original(..., values, ...): ... values.pop() db_operation() ... Note that in this case when db_operation() raises an exception which causes a retry, the second invocation of the function is passed values which has already been modified by the first. Modifying argument data is a generally bad idea unless it's the explicit purpose of the function, but I suspect the combination with a retry wrapper is particularly likely to be overlooked by tests. If anybody would like to review my specific patch it's here: https://review.opendev.org/#/c/658845/ . This is from a reproducible (on a large deployment under load) customer issue, btw, so it isn't theoretical. Matt -- Matthew Booth Red Hat OpenStack Engineer, Compute DFG Phone: +442070094448 (UK)

1 0

Help needed to Support Multi-attach feature
by RAI, SNEHA 14 May '19

14 May '19

Hi Team, I am currently working on multiattach feature for HPE 3PAR cinder driver. For this, while setting up devstack(on stable/queens) I made below change in the local.conf [[local|localrc]] ENABLE_VOLUME_MULTIATTACH=True ENABLE_UBUNTU_CLOUD_ARCHIVE=False /etc/cinder/cinder.conf: [3pariscsi_1] hpe3par_api_url = https://192.168.1.7:8080/api/v1 hpe3par_username = user hpe3par_password = password san_ip = 192.168.1.7 san_login = user san_password = password volume_backend_name = 3pariscsi_1 hpe3par_cpg = my_cpg hpe3par_iscsi_ips = 192.168.11.2,192.168.11.3 volume_driver = cinder.volume.drivers.hpe.hpe_3par_iscsi.HPE3PARISCSIDriver hpe3par_iscsi_chap_enabled = True hpe3par_debug = True image_volume_cache_enabled = True /etc/cinder/policy.json: 'volume:multiattach': 'rule:admin_or_owner' Added https://review.opendev.org/#/c/560067/2/cinder/volume/drivers/hpe/hpe_3par_… change in the code. But I am getting below error in the nova log: Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [None req-2cda6e90-fd45-4bfe-960a-7fca9ba4abab demo admin] [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] Instance failed block device setup: MultiattachNotSupportedByVirtDriver: Volume dc25f09a-6ae1-4b06-a814-73a8afaba62f has 'multiattach' set, which is not supported for this instance. Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] Traceback (most recent call last): Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/compute/manager.py", line 1615, in _prep_block_device Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] wait_func=self._await_block_device_map_created) Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 840, in attach_block_devices Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] _log_and_attach(device) Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 837, in _log_and_attach Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] bdm.attach(*attach_args, **attach_kwargs) Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 46, in wrapped Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] ret_val = method(obj, context, *args, **kwargs) Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 620, in attach Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] virt_driver, do_driver_attach) Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] return f(*args, **kwargs) Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 617, in _do_locked_attach Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] self._do_attach(*args, **_kwargs) Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 602, in _do_attach Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] do_driver_attach) Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] File "/opt/stack/nova/nova/virt/block_device.py", line 509, in _volume_attach Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] volume_id=volume_id) Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] MultiattachNotSupportedByVirtDriver: Volume dc25f09a-6ae1-4b06-a814-73a8afaba62f has 'multiattach' set, which is not supported for this instance. Apr 29 04:23:04 CSSOSBE04-B09 nova-compute[31396]: ERROR nova.compute.manager [instance: fcaa5a47-fc48-489d-9827-6533bfd1a9fa] Apr 29 05:41:20 CSSOSBE04-B09 nova-compute[20455]: DEBUG nova.virt.libvirt.driver [-] Volume multiattach is not supported based on current versions of QEMU and libvirt. QEMU must be less than 2.10 or libvirt must be greater than or equal to 3.10. {{(pid=20455) _set_multiattach_support /opt/stack/nova/nova/virt/libvirt/driver.py:619}} stack@CSSOSBE04-B09:/tmp$ virsh --version 3.6.0 stack@CSSOSBE04-B09:/tmp$ kvm --version QEMU emulator version 2.10.1(Debian 1:2.10+dfsg-0ubuntu3.8~cloud1) Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers openstack volume show -c multiattach -c status sneha1 +-------------+-----------+ | Field | Value | +-------------+-----------+ | multiattach | True | | status | available | +-------------+-----------+ cinder extra-specs-list +--------------------------------------+-------------+--------------------------------------------------------------------+ | ID | Name | extra_specs | +--------------------------------------+-------------+--------------------------------------------------------------------+ | bd077fde-51c3-4581-80d5-5855e8ab2f6b | 3pariscsi_1 | {'volume_backend_name': '3pariscsi_1', 'multiattach': '<is> True'}| +--------------------------------------+-------------+--------------------------------------------------------------------+ echo $OS_COMPUTE_API_VERSION 2.60 pip list | grep python-novaclient DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. python-novaclient 13.0.0 How do I fix this version issue on my setup to proceed? Please help. Thanks & Regards, Sneha Rai

3 8

[publiccloud-wg][publiccloud-sig][telemetry] OpenStack billing initiative for public and private clouds
by Tobias Rydberg 14 May '19

14 May '19

Hi all, During the Denver 2019 Summit the Public Cloud SIG identified that the current situation around public cloud (and private) billing is somewhat fragmented. Some operators use ceilometer, some Cloudkitty, and many use custom implementations. We have started an initiative to find a solution for this issue and figure out a way moving forward that can solve the needs and gaps in current solutions that exists, a solution that will make it easier for all openstack clouds, private and public, to manage billing. We are at a very early stage here, brainstorming around requirements and ideas, and would like to collect as much information about potential requirements and solutions as possible before we get any further. An etherpad [0] is created for the initial collection of information, and we would appreciate feedback and ideas there before the next Public Cloud SIG meeting that will take place 23rd of May 2019 at 1400 UTC in the #openstack-publiccloud channel. Cheers, Tobias [0] https://etherpad.openstack.org/p/publiccloud-sig-billing-implementation-pro… -- Tobias Rydberg Senior Developer Twitter & IRC: tobberydberg www.citynetwork.eu | www.citycloud.com INNOVATION THROUGH OPEN IT INFRASTRUCTURE ISO 9001, 14001, 27001, 27015 & 27018 CERTIFIED

1 0

[senlin][zaqar][networking-midonet][networking-odl][vmware-nsx] Neutron-LBaaS retirement warning!
by Adam Harwell 14 May '19

14 May '19

As you are hopefully already aware, the Neutron-LBaaS project is being retired this cycle (and a lot of the patches to accomplish this will land in the next few days). >From a quick code search, it seems the following projects still include neutron-lbaas in their zuul job configs: networking-odl networking-midonet senlin vmware-nsx zaqar For projects on this list, the retirement of neutron-lbaas *will* cause your zuul jobs to fail. *Please take action to remove this requirement!* It is possible that it is simply an extra unused requirement, but if your project is actually using neutron-lbaas to create loadbalancers, it will be necessary to convert to Octavia. If you need assistance with this change or have any questions, don't hesitate to stop by #openstack-lbaas on IRC and we can help! --Adam Harwell

4 7

[tc] [ceilometer] Rocky release notes for Ceilometer
by Bernd Bausch 14 May '19

14 May '19

I don’t know if it’s correct to tag this message with [tc]; sorry if not. The Ceilometer Rocky release notes are not published on the releases web site [1]. They were distributed via email only [2], it would seem. Whoever manages the releases site or the Ceilometer release notes may want to fill that gap to make it easier to find information. [1] https://releases.openstack.org/rocky/index.html [2] http://lists.openstack.org/pipermail/release-announce/2018-July.txt Bernd

4 5

[dev][keystone] Office hours revamp
by Colleen Murphy 14 May '19

14 May '19

Hi team, One of the outcomes of the PTG was to make more productive use of office hours by designating a topic ahead of time and working through it as a team. I've proposed to officially add the office hour to eavesdrop[1] and started a topic etherpad[2]. This week we'll finish the liaison review if it's not already completed during the meeting, and we'll do some bug triage. We should be sure to continuously reflect on how this is going and whether we should change our process or reschedule the session. Colleen [1] https://review.opendev.org/658909 [2] https://etherpad.openstack.org/p/keystone-office-hours-topics

1 0

[nova][ptg] main etherpad backup
by Sean Mooney 14 May '19

14 May '19

so it looks like the https://etherpad.openstack.org/p/nova-ptg-train-5 etherpad has died like the 4 before it. attached is a offline copy i took near the end of ptg which should have the majoriy fo the content for those that are looking for it the downside is this is just a copy past i did in to a text file so i dont have any of the strike thouughs or autor info in it but all the #Aggree: and other notes we took should still be there. regards sean

1 0

[dev][keystone] Cross-project liaison review
by Colleen Murphy 14 May '19

14 May '19

Hi everyone, I had scheduled time to review our liaison list during the PTG but we decided to defer it until after the PTG so that people who could not attend the PTG had the option of participating. We'll go over the list at the next keystone meeting (Tuesday, 14 May at 1600 UTC in #openstack-meeting-alt), possibly spilling over into the keystone office hours. If you can't make the meeting, please review the current list[1][2] and let me know if you want to update your liaison status or take on any liaison duties. Colleen [1] https://etherpad.openstack.org/p/keystone-liaison-review-2019 [2] https://wiki.openstack.org/wiki/CrossProjectLiaisons

1 0

[neutron] CI meeting on 14th May cancelled
by Slawomir Kaplonski 14 May '19

14 May '19

Hi, I can’t run CI meeting tomorrow (14.05.2019). As Miguel can’t run it also, lets cancel meeting this week. We will meet next week (21.05) as usual. — Slawek Kaplonski Senior software engineer Red Hat

1 0