March 2021 - openstack-discuss - lists.openstack.org

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 04 Aug '22

04 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

5 7

Nova not updating to new size of an extended in-use / attached cinder volume (Ceph RBD) to guest
by Christian Rohmann 28 Jun '22

28 Jun '22

Hello Openstack-Discuss, after some digging I found out that cinder learned to resize attached (in-use) volumes quite a while ago with the introduction of the initial "extend 'in-use' volume" feature: https://review.opendev.org/c/openstack/cinder/+/454287/ The support was then extended to also cover Ceph RBD backed volumes with: https://review.opendev.org/c/openstack/nova/+/613039/ Since this is only about the cinder part, I was wondering if nova would ever find out and would actively rescan the device / volume and quintessentially present an increased new size to the guest. Apparently this is where a certain volume-extended event comes into play: https://review.opendev.org/c/openstack/nova/+/454322/ that is to be emitted by cinder when a volume had been extened. I then went head and tried this with the openstack cli (as Horizon does not seem to support / offer to resize in-use volumes. I run Openstack Train with Ceph RBD as storage. First I ran into an issue with the openstack cli (https://bugs.launchpad.net/cinder/+bug/1871759/comments/2 <https://bugs.launchpad.net/cinder/+bug/1871759/comments/2>) but using cinder extend $volumeid $newsize I was able to resize in-use volumes just fine. The only thing missing was the propagation to the guest. I played around with SCSI rescans, but that did not work, reboots also failed to do anyting in this regards. Stopping and starting the VM did work, but why having the capability to online-resize an attached volume when not being able to resize the guest volume as well? Apparently there seems to be an old and somewhat similar observation / bug with https://bugs.launchpad.net/nova/+bug/1369465 So I was simply wondering if this is expected to be working? Are there any special settings / options I need to set to enable this feature? Thanks and with kind regards Christian

4 9

[nova][dev] Revisiting qemu emulation where guest arch != host arch
by Apsey, Christopher 05 Jan '22

05 Jan '22

All, A few years ago I asked a question[1] about why nova, when given a hw_architecture property from glance for an image, would not end up using the correct qemu-system-xx binary when starting the guest process on a compute node if that compute nodes architecture did not match the proposed guest architecture. As an example, if we had all x86 hosts, but wanted to run an emulated ppc guest, we should be able to do that given that at least one compute node had qemu-system-ppc already installed and libvirt was successfully reporting that as a supported architecture to nova. It seemed like a heavy lift at the time, so it was put on the back burner. I am now in a position to fund a contract developer to make this happen, so the question is: would this be a useful blueprint that would potentially be accepted? Most of the time when people want to run an emulated guest they would just nest it inside of an already running guest of the native architecture, but that severely limits observability and the task of managing any more than a handful of instances in this manner quickly becomes a tangled nightmare of networking, etc. I see real benefit in allowing this scenario to run natively so all of the tooling that exists for fleet management 'just works'. This would also be a significant differentiator for OpenStack as a whole. Thoughts? [1] http://lists.openstack.org/pipermail/openstack-operators/2018-August/015653… Chris Apsey Director | Georgia Cyber Range GEORGIA CYBER CENTER 100 Grace Hopper Lane | Augusta, Georgia | 30901 https://www.gacybercenter.org

2 2

Help with eventlet 0.26.1 and dnspython >= 2
by Thomas Goirand 04 Oct '21

04 Oct '21

Hi, As you may know, eventlet is incompatible with dnspython >= 2.0.0.0rc1. See [1] for the details. However, Debian unstable has 2.0.0. Would there be some good soul willing to help me fix this situation? I would need a patch to fix this, but I'm really not sure how to start. Cheers, Thomas Goirand (zigo)

8 15

[all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community
by Liye Pang(逄立业) 10 Sep '21

10 Sep '21

Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects and that has contributed to the OpenStack community. The following is an introduction to “Venus”. If there is interest in the community, we are interested in proposing it to become an official OpenStack project in the future. Background In the day-to-day operation and maintenance of large-scale cloud platform, the following problems are encountered： l Time-consuming for log querying while the server increasing to thousands. l Difficult to retrieve logs, since there are many modules in the platform, e.g. systems service, compute, storage, network and other platform services. l The large amount and dispersion of log make faults are difficult to be discovered. l Because of distributed and interaction between components of the cloud platform, and scattered logs between components, it will take more time to locate problems. About Venus According to the key requirements of OpenStack in log storage, retrieval, analysis and so on, we introduced Venus project, a unified log management module. This module can provide a one-stop solution to log collection, cleaning, indexing, analysis, alarm, visualization, report generation and other needs, which involves helping operator or maintainer to quickly solve retrieve problems, grasp the operational health of the platform, and improve the management capabilities of the cloud platform. Additionally, this module plans to use machine learning algorithms to quickly locate IT failures and root causes, and improve operation and maintenance efficiency. Application scenario Venus played a key role in the following scenarios： l Retrieval: Provide a simple and easy-to-use way to retrieve all log and the context. l Analysis: Realize log association, field value statistics, and provide multi-scene and multi-dimensional visual analysis reports. l Alerts：Convert retrieval into active alerts to realize the error finding in massive logs. l Issue location: Establish a chain relationship and knowledge graphs to quickly locate problems. Overall structure The architecture of log management system based on Venus and elastic search is as follows: Diagram 0: Architecture of Venus venus_api: API module，provide API、rest-api service. venus_manager: Internal timing task module to realize the core functions of the log system. Current progress The current progress of the Venus project is as follows: l Collection：Develop fluentd collection tasks based on collectd to read, filter, format and send plug-ins for OpenStack, operating systems, and platform services, etc. l Index：Dealing with multi-dimensional index data in elasticsearch, and provide more concise and comprehensive authentication interface to return query results. l Analysis：Analyzing and display the related module errors, Mariadb connection errors, and Rabbitmq connection errors. l Alerts：Develop alarm task code to set threshold for the number of error logs of different modules at different times, and provides alarm services and notification services. l Location：Develop the call chain analysis function based on global_requested series, which can show the execution sequence, time and error information, etc., and provide the export operation. l Management：Develop configuration management functions in the log system, such as alarm threshold setting, timing task management, and log saving time setting, etc. Application examples Two examples of Venus application scenarios are as follows. 1. The virtual machine creation operation was performed on the cloud platform and it was found that the virtual machine was not created successfully. First, we can find the request id of the operation and jump to the virtual machine creation call chain page. Then, we can query the calling process, view and download the details of the log of the call. 2. In the cloud platform, the error log of each module can be converted into alarms to remind the users. Further, we can retrieve the details of the error log and error log statistics. Next step The next step of the Venus project is as follows: * Collection：In addition to fluent, other collection plugins such as logstash will be integrated. * Analysis: Explore more operation and maintenance scenarios, and conduct statistical analysis and alarm on key data. * display: The configuration, analysis and alarm of Venus will be integrated into horizon in the form of plugin. * location: Form clustering log and construct knowledge map, and integrate algorithm class library to locate the root cause of the fault. Venus Project Registry Venus library: https://opendev.org/inspur/venus You can grab the source code using the following git command: git clone https://opendev.org/inspur/venus.git Venus Demo Youtu.be: <https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM

7 15

Question regarding the production status of DB2 in Trove
by Bekir Fajkovic 31 May '21

31 May '21

Hello! One question regarding the development status of DB2 inside Trove. I see that DB2 is still in experimental phase, when could it be expected for that datastore type to get production ready status? Best Regards. Bekir Fajkovic Senior DBA Mobile: +46 70 019 48 47 www.citynetwork.eu | www.citycloud.com INNOVATION THROUGH OPEN IT INFRASTRUCTURE ISO 9001, 14001, 27001, 27015 & 27018 CERTIFIED

3 8

[simplification] Making ask.openstack.org read-only
by Thierry Carrez 27 May '21

27 May '21

Hi everyone, This has been discussed several times on this mailing list in the past, but we never got to actually pull the plug. Ask.openstack.org was launched in 2013. The reason for hosting our own setup was to be able to support multiple languages, while StackOverflow rejected our proposal to have our own openstack-branded StackExchange site. The Chinese ask.o.o side never really took off. The English side also never really worked perfectly (like email alerts are hopelessly broken), but we figured it would get better with time if a big community formed around it. Fast-forward to 2020 and the instance is lacking volunteers to help run it, while the code (and our customization of it) has become more complicated to maintain. It regularly fails one way or another, and questions there often go unanswered, making us look bad. Of the top 30 users, most have abandoned the platform since 2017, leaving only Bernd Bausch actively engaging and helping moderate questions lately. We have called for volunteers several times, but the offers for help never really materialized. At the same time, people are asking OpenStack questions on StackOverflow, and sometimes getting answers there[1]. The fragmentation of the "questions" space is not helping users getting good answers. I think it's time to pull the plug, make ask.openstack.org read-only (so that links to old answers are not lost) and redirect users to the mailing-list and the "OpenStack" tag on StackOverflow. I picked StackOverflow since it seems to have the most openstack questions (2,574 on SO, 76 on SuperUser and 430 on ServerFault). We discussed that option several times, but I now proposed a change to actually make it happen: https://review.opendev.org/#/c/746497/ It's always a difficult decision to make to kill a resource, but I feel like in this case, consolidation and simplification would help. Thoughts, comments? [1] https://stackoverflow.com/questions/tagged/openstack -- Thierry

10 18

[stein][neutron] gratuitous arp
by Ignazio Cassano 14 May '21

14 May '21

Hello, I have a problem on stein neutron. When a vm migrate from one node to another I cannot ping it for several minutes. If in the vm I put a script that ping the gateway continously, the live migration works fine and I can ping it. Why this happens ? I read something about gratuitous arp. Please, help me ? Any workaround , please ? Best Regards Ignazio

3 33

[octavia] victoria - loadbalancer works but its operational status is offline
by apps＠mossakowski.ch 13 May '21

13 May '21

Hello, I have stable/victoria baremetal openstack with octavia installed on centos8 using openvswitch mechanism driver: octavia api on controller, health-manager,housekeeping,worker on 3 compute/network nodes. Official docs include only ubuntu with linuxbridge mechanism but I used https://github.com/prastamaha/openstack-octavia as a reference to get it working on centos8 with ovs. I will push those docs instructions for centos8 soon: https://github.com/openstack/octavia/tree/master/doc/source/install. I created basic http scenario using https://docs.openstack.org/octavia/victoria/user/guides/basic-cookbook.html…. Loadbalancer works but its operational status is offline (openstack_loadbalancer_outputs.txt). On all octavia workers I see the same warning message in health_manager.log: Health Manager experienced an exception processing a heartbeat message from ('172.31.255.233', 1907). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode' I've searched for related active bug but all I found is this not related in my opinion: https://storyboard.openstack.org/#!/story/2008615 I'm attaching all info I've gathered: - octavia.conf and health_manager debug logs (octavia_config_and_health_manager_logs.txt) - tcpdump from amphora VM (tcpdump_from_amphora_vm.txt) - tcpdump from octavia worker (tcpdump_from_octavia_worker.txt) - debug amphora-agent.log from amphora VM (amphora-agent.log) Can you point me to the right direction what I have missed? Thanks! Piotr Mossakowski https://github.com/moss2k13

4 5

[ironic] Announcing deprecation of the iSCSI deploy interface
by Dmitry Tantsur 04 May '21

04 May '21

Hi all, Following up to the previous mailing list [1] and virtual meetup [2] discussions, I would like to announce the plans to deprecate the 'iscsi' deploy interface. This is the updated plan discussed on the virtual meetup: 1) In the Victoria cycle (i.e. right now): - Fill in the detected feature gaps [3]. - Switch off the iscsi deploy interface by default. - Change [agent]image_dowload_source to HTTP by default. - Give the direct deploy a higher priority, so that it's used by default unless disabled. - Mark it as deprecated in the code (causing warnings when enabled). - Release a major version of ironic to highlight the defaults changes. 2) In the W cycle: - Keep the iscsi deploy deprecated. - Listen to operators' feedback. 3) In the X cycle - Remove the iscsi deploy completely from ironic and IPA. - Remove support code from ironic-lib with a major version bump. Please let us know if you have any questions or concerns. Dmitry [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016681.h… [2] https://etherpad.opendev.org/p/Ironic-Victoria-midcycle [3] https://storyboard.openstack.org/#!/story/2008075 -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

1 1

[release] Meeting Time Poll
by Herve Beraud 03 May '21

03 May '21

Hello We have a few regular attendees of the Release Management meeting who have conflicts with the current meeting time. As a result, we would like to find a new time to hold the meeting. I've created a Doodle poll[1] for everyone to give their input on times. It's mostly limited to times that reasonably overlap the working day in the US and Europe since that's where most of our attendees are located. If you attend the Release Management meeting, please fill out the poll so we can hopefully find a time that works better for everyone. For the sake of organization and to allow everyone to schedule his agenda accordingly, the poll will be closed on April 5th. On that date, I will announce the time of this meeting and the date on which it will take effect. Thanks! [1] https://doodle.com/poll/ip6tg4fvznz7p3qx -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 5

[barbican][oslo][nova][glance][cinder] cursive library status
by Brian Rosmaita 03 May '21

03 May '21

Hello Barbican team, Apologies for not including barbican in the previous thread on this topic: http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019430… The situation is that cursive is used by Nova, Glance, and Cinder and we'd like to move it out of the 'x' namespace into openstack governance. The question is then what team would oversee it. It seems like a good fit for Oslo, and the Oslo team seems OK with that, but since barbican-core is currently included in cursive-core, it make sense to give the Barbican team first dibs. From the consuming teams' side, I don't think we have a preference as long as it's clear who we need to bother about approvals if a bugfix is posted for review. Thus my ask is that the Barbican team indicate whether they'd like to move cursive to the 'openstack' namespace under their governance, or whether they'd prefer Oslo to oversee the library. Thank you! brian

5 5

[Vitrage] Cannot delete old entities on graph
by Adam Tomas 27 Apr '21

27 Apr '21

Hi, I’ve deployed Kolla-Ansible (Victoria) and after using it for some time I have errors from Vitrage service: 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init [-] Got Exception for event {'vitrage_entity_type': 'consistency', 'vitrage_datasource_action': 'update', 'vitrage_sample_date': '2021-03-18T13:30:19Z', 'vitrage_event_type': 'delete_entity', 'vitrage_id': 'a19330f1-49c1-4a97-9787-c80b645f51a6', 'id': '93c1df77-dbb5-4ba7-9b7d-16fad97169f1', 'vitrage_type': 'cinder.volume', 'vitrage_category': 'RESOURCE', 'is_real_vitrage_id': True}: KeyError: 'vitrage_is_placeholder' 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init Traceback (most recent call last): 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/entity_graph/graph_init.py", line 146, in do_work 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init return do_work_func(event) 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/entity_graph/graph_init.py", line 111, in process_event 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init self.processor.process_event(event) 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/entity_graph/processor/processor.py", line 63, in process_event 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init self.actions[entity.action](entity.vertex, entity.neighbors) 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/entity_graph/processor/processor.py", line 149, in delete_entity 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init PUtils.delete_placeholder_vertex(self.entity_graph, vertex) 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/entity_graph/processor/processor_utils.py", line 68, in delete_placeholder_vertex 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init if not vertex[VProps.VITRAGE_IS_PLACEHOLDER]: 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/graph/driver/elements.py", line 24, in __getitem__ 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init return self.properties[key] 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init KeyError: 'vitrage_is_placeholder' 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init As I can see in Horizon, there are some old volumes , which in Cinder database have proper (DELETED/DETACHED) status, but they’re still visible on Entity Graph, and as in error message above - cannot be deleted by vitrage-graph. How to delete such entities? Best regards, Adam Tomas

2 7

Create OpenStack VMs in few seconds
by open infra 23 Apr '21

23 Apr '21

Hi, I was looking for a way to provision and connect to VMs within a few seconds and ended up in the OpenStack Summit demo on 'OpenStack Virtual Machine Quickly Available in One Second'. I highly appreciate if someone can provide a tutorial or related documentation on how to achieve this? Regards, Danishka

6 16

[kuryr] vPTG April 2021
by Maysa De Macedo Souza 23 Apr '21

23 Apr '21

Hello, Two sessions were scheduled for Kuryr on the upcoming PTG: - 7-8 UTC on April 20 - 2-3 UTC on April 22 Everyone is more than welcome to join the sessions and check our future plans, give feedback or discuss anything regarding Kuryr. Even though participation is free registration is needed[1]. Regards, Maysa Macedo. [1] https://april2021-ptg.eventbrite.com

2 2

[nova] Nominate sean-k-mooney for nova-specs-core
by Stephen Finucane 19 Apr '21

19 Apr '21

Hey, Sean has been working on nova for what seems like yonks now. Each cycle, they spend a significant amount of time reviewing proposed specs and contributing to discussions at the PTG. This is important work and their contributions provide everyone with a deep pool of knowledge on all things networking and hardware upon which to draw. I think the nova project would benefit from their addition to the specs core reviewer team and I therefore propose we add Sean to nova- specs-core. Assuming there are no objections, I'll work with gibi to add Sean to nova-specs- core next week. Cheers, Stephen

9 9

[Murano] Please help with murano image creation
by Bujack, Stefan 19 Apr '21

19 Apr '21

Hello, we want to create the murano image for an openstack victoria deployment. We tried to follow this documentation: https://docs.openstack.org/murano/victoria/reference/appendix/articles/imag… But we were not able to successfully create an image. At first we got the following error: murano-agent requires Python '>=3.6' but the running Python is 2.7.17 Then I changed the release to focal and got another error: Unable to locate package python-pip Is anybody so kind to help me with this problem? Many thanks in advance Greets Stefan Bujack root@kolla-2:~# cat /etc/issue Ubuntu 20.04.2 LTS \n \l root@kolla-2:~# export GITDIR=~/git root@kolla-2:~# mkdir -p $GITDIR root@kolla-2:~# cd $GITDIR root@kolla-2:~/git# git clone https://opendev.org/openstack/murano Cloning into 'murano'... remote: Enumerating objects: 34335, done. remote: Counting objects: 100% (34335/34335), done. remote: Compressing objects: 100% (14856/14856), done. remote: Total 34335 (delta 22813), reused 29217 (delta 18268) Receiving objects: 100% (34335/34335), 16.73 MiB | 7.42 MiB/s, done. Resolving deltas: 100% (22813/22813), done. root@kolla-2:~/git# git clone https://opendev.org/openstack/murano-agent Cloning into 'murano-agent'... remote: Enumerating objects: 6696, done. remote: Counting objects: 100% (6696/6696), done. remote: Compressing objects: 100% (2792/2792), done. remote: Total 6696 (delta 3766), reused 6475 (delta 3586) Receiving objects: 100% (6696/6696), 4.98 MiB | 5.46 MiB/s, done. Resolving deltas: 100% (3766/3766), done. root@kolla-2:~/git# pip3 install diskimage-builder Collecting diskimage-builder Downloading diskimage_builder-3.7.0-py3-none-any.whl (608 kB) |████████████████████████████████| 608 kB 4.8 MB/s Requirement already satisfied: stevedore>=1.20.0 in /usr/lib/python3/dist-packages (from diskimage-builder) (1.32.0) Collecting flake8<4.0.0,>=3.6.0 Downloading flake8-3.8.4-py2.py3-none-any.whl (72 kB) |████████████████████████████████| 72 kB 1.6 MB/s Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /usr/lib/python3/dist-packages (from diskimage-builder) (5.4.5) Collecting networkx>=1.10 Downloading networkx-2.5-py3-none-any.whl (1.6 MB) |████████████████████████████████| 1.6 MB 7.6 MB/s Requirement already satisfied: PyYAML>=3.12 in /usr/lib/python3/dist-packages (from diskimage-builder) (5.3.1) Collecting pycodestyle<2.7.0,>=2.6.0a1 Downloading pycodestyle-2.6.0-py2.py3-none-any.whl (41 kB) |████████████████████████████████| 41 kB 331 kB/s Collecting pyflakes<2.3.0,>=2.2.0 Downloading pyflakes-2.2.0-py2.py3-none-any.whl (66 kB) |████████████████████████████████| 66 kB 5.8 MB/s Collecting mccabe<0.7.0,>=0.6.0 Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB) Requirement already satisfied: decorator>=4.3.0 in /usr/lib/python3/dist-packages (from networkx>=1.10->diskimage-builder) (4.4.2) Installing collected packages: pycodestyle, pyflakes, mccabe, flake8, networkx, diskimage-builder Successfully installed diskimage-builder-3.7.0 flake8-3.8.4 mccabe-0.6.1 networkx-2.5 pycodestyle-2.6.0 pyflakes-2.2.0 root@kolla-2:~/git# apt install qemu-utils curl tox Reading package lists... Done Building dependency tree Reading state information... Done curl is already the newest version (7.68.0-1ubuntu2.4). curl set to manually installed. The following additional packages will be installed: ibverbs-providers javascript-common libboost-iostreams1.71.0 libboost-thread1.71.0 libibverbs1 libiscsi7 libjs-jquery libjs-sphinxdoc libjs-underscore libnl-route-3-200 librados2 librbd1 librdmacm1 python3-distlib python3-filelock python3-pluggy python3-py python3-toml python3-virtualenv qemu-block-extra sharutils virtualenv Suggested packages: apache2 | lighttpd | httpd subversion python3-pytest debootstrap sharutils-doc bsd-mailx | mailx The following NEW packages will be installed: ibverbs-providers javascript-common libboost-iostreams1.71.0 libboost-thread1.71.0 libibverbs1 libiscsi7 libjs-jquery libjs-sphinxdoc libjs-underscore libnl-route-3-200 librados2 librbd1 librdmacm1 python3-distlib python3-filelock python3-pluggy python3-py python3-toml python3-virtualenv qemu-block-extra qemu-utils sharutils tox virtualenv 0 upgraded, 24 newly installed, 0 to remove and 14 not upgraded. Need to get 8108 kB of archives. After this operation, 37.8 MB of additional disk space will be used. Do you want to continue? [Y/n] y Get:1 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libnl-route-3-200 amd64 3.4.0-1 [149 kB] Get:2 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libibverbs1 amd64 28.0-1ubuntu1 [53.6 kB] Get:3 http://de.archive.ubuntu.com/ubuntu focal/main amd64 ibverbs-providers amd64 28.0-1ubuntu1 [232 kB] Get:4 http://de.archive.ubuntu.com/ubuntu focal/main amd64 javascript-common all 11 [6066 B] Get:5 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libboost-iostreams1.71.0 amd64 1.71.0-6ubuntu6 [237 kB] Get:6 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libboost-thread1.71.0 amd64 1.71.0-6ubuntu6 [249 kB] Get:7 http://de.archive.ubuntu.com/ubuntu focal/main amd64 librdmacm1 amd64 28.0-1ubuntu1 [64.9 kB] Get:8 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libiscsi7 amd64 1.18.0-2 [63.9 kB] Get:9 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libjs-jquery all 3.3.1~dfsg-3 [329 kB] Get:10 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libjs-underscore all 1.9.1~dfsg-1 [98.6 kB] Get:11 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] Get:12 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.7-0ubuntu0.20.04.2 [3204 kB] Get:13 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.7-0ubuntu0.20.04.2 [1609 kB] Get:14 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-distlib all 0.3.0-1 [116 kB] Get:15 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-filelock all 3.0.12-2 [7948 B] Get:16 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] Get:17 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-py all 1.8.1-1 [65.4 kB] Get:18 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-toml all 0.10.0-3 [14.6 kB] Get:19 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-virtualenv all 20.0.17-1 [63.4 kB] Get:20 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.12 [53.8 kB] Get:21 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.12 [974 kB] Get:22 http://de.archive.ubuntu.com/ubuntu focal/main amd64 sharutils amd64 1:4.15.2-4build1 [155 kB] Get:23 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 virtualenv all 20.0.17-1 [2132 B] Get:24 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 tox all 3.13.2-2 [244 kB] Fetched 8108 kB in 0s (31.9 MB/s) Selecting previously unselected package libnl-route-3-200:amd64. (Reading database ... 118797 files and directories currently installed.) Preparing to unpack .../00-libnl-route-3-200_3.4.0-1_amd64.deb ... Unpacking libnl-route-3-200:amd64 (3.4.0-1) ... Selecting previously unselected package libibverbs1:amd64. Preparing to unpack .../01-libibverbs1_28.0-1ubuntu1_amd64.deb ... Unpacking libibverbs1:amd64 (28.0-1ubuntu1) ... Selecting previously unselected package ibverbs-providers:amd64. Preparing to unpack .../02-ibverbs-providers_28.0-1ubuntu1_amd64.deb ... Unpacking ibverbs-providers:amd64 (28.0-1ubuntu1) ... Selecting previously unselected package javascript-common. Preparing to unpack .../03-javascript-common_11_all.deb ... Unpacking javascript-common (11) ... Selecting previously unselected package libboost-iostreams1.71.0:amd64. Preparing to unpack .../04-libboost-iostreams1.71.0_1.71.0-6ubuntu6_amd64.deb ... Unpacking libboost-iostreams1.71.0:amd64 (1.71.0-6ubuntu6) ... Selecting previously unselected package libboost-thread1.71.0:amd64. Preparing to unpack .../05-libboost-thread1.71.0_1.71.0-6ubuntu6_amd64.deb ... Unpacking libboost-thread1.71.0:amd64 (1.71.0-6ubuntu6) ... Selecting previously unselected package librdmacm1:amd64. Preparing to unpack .../06-librdmacm1_28.0-1ubuntu1_amd64.deb ... Unpacking librdmacm1:amd64 (28.0-1ubuntu1) ... Selecting previously unselected package libiscsi7:amd64. Preparing to unpack .../07-libiscsi7_1.18.0-2_amd64.deb ... Unpacking libiscsi7:amd64 (1.18.0-2) ... Selecting previously unselected package libjs-jquery. Preparing to unpack .../08-libjs-jquery_3.3.1~dfsg-3_all.deb ... Unpacking libjs-jquery (3.3.1~dfsg-3) ... Selecting previously unselected package libjs-underscore. Preparing to unpack .../09-libjs-underscore_1.9.1~dfsg-1_all.deb ... Unpacking libjs-underscore (1.9.1~dfsg-1) ... Selecting previously unselected package libjs-sphinxdoc. Preparing to unpack .../10-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... Selecting previously unselected package librados2. Preparing to unpack .../11-librados2_15.2.7-0ubuntu0.20.04.2_amd64.deb ... Unpacking librados2 (15.2.7-0ubuntu0.20.04.2) ... Selecting previously unselected package librbd1. Preparing to unpack .../12-librbd1_15.2.7-0ubuntu0.20.04.2_amd64.deb ... Unpacking librbd1 (15.2.7-0ubuntu0.20.04.2) ... Selecting previously unselected package python3-distlib. Preparing to unpack .../13-python3-distlib_0.3.0-1_all.deb ... Unpacking python3-distlib (0.3.0-1) ... Selecting previously unselected package python3-filelock. Preparing to unpack .../14-python3-filelock_3.0.12-2_all.deb ... Unpacking python3-filelock (3.0.12-2) ... Selecting previously unselected package python3-pluggy. Preparing to unpack .../15-python3-pluggy_0.13.0-2_all.deb ... Unpacking python3-pluggy (0.13.0-2) ... Selecting previously unselected package python3-py. Preparing to unpack .../16-python3-py_1.8.1-1_all.deb ... Unpacking python3-py (1.8.1-1) ... Selecting previously unselected package python3-toml. Preparing to unpack .../17-python3-toml_0.10.0-3_all.deb ... Unpacking python3-toml (0.10.0-3) ... Selecting previously unselected package python3-virtualenv. Preparing to unpack .../18-python3-virtualenv_20.0.17-1_all.deb ... Unpacking python3-virtualenv (20.0.17-1) ... Selecting previously unselected package qemu-block-extra:amd64. Preparing to unpack .../19-qemu-block-extra_1%3a4.2-3ubuntu6.12_amd64.deb ... Unpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.12) ... Selecting previously unselected package qemu-utils. Preparing to unpack .../20-qemu-utils_1%3a4.2-3ubuntu6.12_amd64.deb ... Unpacking qemu-utils (1:4.2-3ubuntu6.12) ... Selecting previously unselected package sharutils. Preparing to unpack .../21-sharutils_1%3a4.15.2-4build1_amd64.deb ... Unpacking sharutils (1:4.15.2-4build1) ... Selecting previously unselected package virtualenv. Preparing to unpack .../22-virtualenv_20.0.17-1_all.deb ... Unpacking virtualenv (20.0.17-1) ... Selecting previously unselected package tox. Preparing to unpack .../23-tox_3.13.2-2_all.deb ... Unpacking tox (3.13.2-2) ... Setting up javascript-common (11) ... Setting up python3-filelock (3.0.12-2) ... Setting up python3-py (1.8.1-1) ... Setting up python3-distlib (0.3.0-1) ... Setting up libboost-iostreams1.71.0:amd64 (1.71.0-6ubuntu6) ... Setting up libnl-route-3-200:amd64 (3.4.0-1) ... Setting up python3-toml (0.10.0-3) ... Setting up python3-pluggy (0.13.0-2) ... Setting up libboost-thread1.71.0:amd64 (1.71.0-6ubuntu6) ... Setting up libjs-jquery (3.3.1~dfsg-3) ... Setting up sharutils (1:4.15.2-4build1) ... Setting up libjs-underscore (1.9.1~dfsg-1) ... Setting up libibverbs1:amd64 (28.0-1ubuntu1) ... Setting up ibverbs-providers:amd64 (28.0-1ubuntu1) ... Setting up python3-virtualenv (20.0.17-1) ... Setting up virtualenv (20.0.17-1) ... Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... Setting up librdmacm1:amd64 (28.0-1ubuntu1) ... Setting up librados2 (15.2.7-0ubuntu0.20.04.2) ... Setting up tox (3.13.2-2) ... Setting up librbd1 (15.2.7-0ubuntu0.20.04.2) ... Setting up libiscsi7:amd64 (1.18.0-2) ... Setting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.12) ... Setting up qemu-utils (1:4.2-3ubuntu6.12) ... Processing triggers for libc-bin (2.31-0ubuntu9.2) ... Processing triggers for man-db (2.9.1-1) ... Processing triggers for install-info (6.7.0.dfsg.2-5) ... root@kolla-2:~/git# export ELEMENTS_PATH=$GITDIR/murano/contrib/elements:$GITDIR/murano-agent/contrib/elements root@kolla-2:~/git# disk-image-create vm ubuntu murano-agent -o murano-agent.qcow2 2021-02-18 12:52:34.387 | murano-agent requires Python '>=3.6' but the running Python is 2.7.17 root@kolla-2:~/git# export DIB_RELEASE=focal root@kolla-2:~/git# disk-image-create vm ubuntu murano-agent -o murano-agent.qcow2 2021-02-18 12:55:41.222 | E: Unable to locate package python-pip

2 1

[kolla-ansible][horizon] policy.yaml/json files
by Adam Tomas 16 Apr '21

16 Apr '21

Hi, Im not quite clear about policy.yaml/json files in kolla-ansible. Let assume, that I need to allow one of project users to add other users to the project. So I create „project_admin” role and assign it to this user. Then I found /etc/kolla/keystone/policy.json.test file, which I use as template. There is rule „identity:create_credential” : „(role:admin and system_scope:all)” so I add „or role:project_admin” and put file in /etc/kolla/config/keystone/ and reconfigure kolla. And now few questions: 1. policy.json (or policy.yaml) always overwrite all default policies? I mean if I only add one rule to this file then other rules will „disappear” or will have default values? Is there any way to only overwrite some default rules and leave rest with defaults? Like with .conf files 2. what about Horizon and visibility of options? In mentioned case putting the same policy.json file in /etc/kolla/config/keystone/ and /etc/kolla/config/horizon/ should „unblock” Add User button for user with project_admin role? Or how to achieve it? 3. does Horizon need the duplicated policy.json files from other services in it’s configuration folder or is it enough to write policy.json for services I want to change? 4. when I assign admin role to a user with projectID (openstack role add —project PROJECT_ID —user SOME_USER admin) this user sees in Horizon everything systemwide, not only inside this project… Which rules should be created to allow him to see only users and resources which belongs to this project? Best regards Adam

2 7

[manila][ptg] Xena PTG Planning
by Goutham Pacha Ravi 15 Apr '21

15 Apr '21

Hello Zorillas and Interested Stackers, As you're aware, the virtual PTG for the Xena release cycle is between April 19-23, 2021. If you haven't registered yet, you must do so as soon as possible! [1]. We've signed up for some slots on the PTG timeslots ethercalc [2]. The PTG Planning etherpad [3] is now live. Please go ahead and add your name/irc nick and propose any topics. You may propose topics even if you wouldn't like to moderate the discussion. Thanks, and hope to see you all there! Goutham [1] https://april2021-ptg.eventbrite.com/ [2] https://ethercalc.net/oz7q0gds9zfi [3] https://etherpad.opendev.org/p/xena-ptg-manila-planning

1 1

[oslo][security-sig] Please revisit your open vulnerability report
by Jeremy Stanley 12 Apr '21

12 Apr '21

Please help the OpenStack Vulnerability Management Team by taking a look at the following report: keystonemiddleware connections to memcached from neutron-server grow beyond configured values https://launchpad.net/bugs/1883659 Can it be exploited by a nefarious actor, and if so, how? Is it likely to be fixable in all our supported stable branches, respecting stable backport policy? What deployment configurations and options might determine whether a particular installation is susceptible? This is the sort of feedback we depend on to make determinations regarding whether and how to keep the public notified, so they can make informed decisions. Thanks for doing your part to keep our users safe! -- Jeremy Stanley

2 6

[ops][cinder][nova] os-brick upcoming releases
by Brian Rosmaita 12 Apr '21

12 Apr '21

Hello operators, You may have heard about a potential data-loss bug [0] that was recently discovered. It has been fixed in the upcoming wallaby release and we are planning to backport to all stable branches and do new os-brick releases from the releasable stable branches. In the meantime, the bug occurs if the multipath configuration option on a compute is changed while volumes are attached to instances on that compute. The possible data loss may occur when the volumes are detached (migration, volume-detach, etc.). Thus, before the new os-brick releases are available, the issue can nonetheless be averted by not making such a configuration change under those circumstances. The new os-brick releases will be: - victoria: 4.0.3 - ussuri: 3.0.6 - train: 2.10.6 The stein, rocky, and queens branches are in Extended Maintenance mode and are no longer released from, but critical fixes are backported to them when possible, though it may take a while before these are merged. [0] https://launchpad.net/bugs/1921381

1 1

[dev][cinder][keystone] Properly consuming system-scope in cinder
by Lance Bragstad 09 Apr '21

09 Apr '21

Hey folks, As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0]. The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1] Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests. In the process of implementing support for tempest testing, I noticed that service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6]. This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7]. Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of the assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more. First, we remove project IDs from cinder's API path. This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot. Second, we update cinder's clients, including tempest, to put the project ID on the URL. After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone. Thoughts? [0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac [1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6] https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3… [7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/

4 8

[puppet] Proposing Alan Bishop (abishop) for puppet-cinder core and puppet-glance core
by Takashi Kajinami 08 Apr '21

08 Apr '21

Hello, I'd like to propose Alan Bishop (abishop) for the core team of puppet-cinder and puppet-glance. Alan has been actively involved in these 2 modules for a few years and has implemented some nice features like multiple backend support in glance, cinder s3 backup driver and etc, which expanded adoption of puppet-openstack. He has also provided good reviews on patches for these 2 repos based on his understanding about our code, puppet and serverspec. He is an active contributor to cinder and has deep knowledge about it. In addition He is also a core review in TripleO, which consumes our puppet modules, and mainly covers storage components like cinder and glance, so he is familiar with the way how these two components are deployed and configured. I believe adding him to our board helps us improve our review of these two modules. I'll wait for one week to hear any feedback from other core reviewers. Thank you, Takashi

6 6