August 2021 - openstack-discuss - lists.openstack.org

[all][tc][goals] Project Specific PTL and Contributor Documentation: Week R-20 Update
by Ghanshyam Mann 17 Sep '21

17 Sep '21

Hello Everyone, As you know, we did not select any community-wide goal for Xena cycle and in PTG, TC decided to spend more time on the previous cycle pending community-wide goal work[1] One of the previous cycle (Ussuri) goals is 'Project Specific PTL and Contributor Documentation' which still not completed by many projects. - https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-c… I am starting the work to finish this goal which is pretty straight forwards and sending the updates in this ML thread. Please review the patches and merge them or push the patches for your project's repo if not yet done. Gerrit Topic: https://review.opendev.org/q/topic:%22project-ptl-and-contrib-docs%22+(stat… Tracking: https://storyboard.openstack.org/#!/story/2007236 Progress Summary: =============== * Projects completed: 32 * Projects for which patches are up and required to merge those: 10 * Projects for which patches are not yet up: 12 [1] https://etherpad.opendev.org/p/tc-xena-tracker (Item#2) -gmann

2 2

[all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community
by Liye Pang(逄立业) 10 Sep '21

10 Sep '21

Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects and that has contributed to the OpenStack community. The following is an introduction to “Venus”. If there is interest in the community, we are interested in proposing it to become an official OpenStack project in the future. Background In the day-to-day operation and maintenance of large-scale cloud platform, the following problems are encountered： l Time-consuming for log querying while the server increasing to thousands. l Difficult to retrieve logs, since there are many modules in the platform, e.g. systems service, compute, storage, network and other platform services. l The large amount and dispersion of log make faults are difficult to be discovered. l Because of distributed and interaction between components of the cloud platform, and scattered logs between components, it will take more time to locate problems. About Venus According to the key requirements of OpenStack in log storage, retrieval, analysis and so on, we introduced Venus project, a unified log management module. This module can provide a one-stop solution to log collection, cleaning, indexing, analysis, alarm, visualization, report generation and other needs, which involves helping operator or maintainer to quickly solve retrieve problems, grasp the operational health of the platform, and improve the management capabilities of the cloud platform. Additionally, this module plans to use machine learning algorithms to quickly locate IT failures and root causes, and improve operation and maintenance efficiency. Application scenario Venus played a key role in the following scenarios： l Retrieval: Provide a simple and easy-to-use way to retrieve all log and the context. l Analysis: Realize log association, field value statistics, and provide multi-scene and multi-dimensional visual analysis reports. l Alerts：Convert retrieval into active alerts to realize the error finding in massive logs. l Issue location: Establish a chain relationship and knowledge graphs to quickly locate problems. Overall structure The architecture of log management system based on Venus and elastic search is as follows: Diagram 0: Architecture of Venus venus_api: API module，provide API、rest-api service. venus_manager: Internal timing task module to realize the core functions of the log system. Current progress The current progress of the Venus project is as follows: l Collection：Develop fluentd collection tasks based on collectd to read, filter, format and send plug-ins for OpenStack, operating systems, and platform services, etc. l Index：Dealing with multi-dimensional index data in elasticsearch, and provide more concise and comprehensive authentication interface to return query results. l Analysis：Analyzing and display the related module errors, Mariadb connection errors, and Rabbitmq connection errors. l Alerts：Develop alarm task code to set threshold for the number of error logs of different modules at different times, and provides alarm services and notification services. l Location：Develop the call chain analysis function based on global_requested series, which can show the execution sequence, time and error information, etc., and provide the export operation. l Management：Develop configuration management functions in the log system, such as alarm threshold setting, timing task management, and log saving time setting, etc. Application examples Two examples of Venus application scenarios are as follows. 1. The virtual machine creation operation was performed on the cloud platform and it was found that the virtual machine was not created successfully. First, we can find the request id of the operation and jump to the virtual machine creation call chain page. Then, we can query the calling process, view and download the details of the log of the call. 2. In the cloud platform, the error log of each module can be converted into alarms to remind the users. Further, we can retrieve the details of the error log and error log statistics. Next step The next step of the Venus project is as follows: * Collection：In addition to fluent, other collection plugins such as logstash will be integrated. * Analysis: Explore more operation and maintenance scenarios, and conduct statistical analysis and alarm on key data. * display: The configuration, analysis and alarm of Venus will be integrated into horizon in the form of plugin. * location: Form clustering log and construct knowledge map, and integrate algorithm class library to locate the root cause of the fault. Venus Project Registry Venus library: https://opendev.org/inspur/venus You can grab the source code using the following git command: git clone https://opendev.org/inspur/venus.git Venus Demo Youtu.be: <https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM

7 15

Tenant vlan Network Issue
by Karera Tony 08 Sep '21

08 Sep '21

Hello Team, Whenever I create an internal network with type vlan. The Instances don't get IPs but for external Network it's fine. Below is the etc/kolla/config/neutron/ml2_conf.ini file [ml2] type_drivers = flat,vlan,vxlan tenant_network_types = vlan,flat mechanism_drivers = openvswitch,l2population [ml2_type_vlan] network_vlan_ranges = physnet1 [ml2_type_flat] flat_networks = physnet1 Is there something I need to change or I need to configure the interface differently Regards Tony Karera

4 12

[keystone] Weekly meetings
by Douglas Mendizabal 07 Sep '21

07 Sep '21

Hi All, I'm helping the Keystone team get the weekly meetings back on track. [1] I'd like to propose a change to the meeting time slot from Tuesday at 1700 UTC to Tuesday at 1500 UTC to make it a little earlier for our contributors in EMEA. I'll be out next week, so I won't be available to chair the meeting, but I should be back the following week for the August 31 meeting. If there are no objections to this time change, I'll go ahead and propose a change to the meetings repo. Thanks, - Douglas Mendizábal (redrobot) [1] https://meetings.opendev.org/#Keystone_Team_Meeting

1 1

Changing Magnum Heat template to create node network with vxlan instead of vlan
by Karera Tony 06 Sep '21

06 Sep '21

Hello Team, I was able to deploy Openstack with Magnum and Zun enabled. Unfortunately, When I create a cluster, Heat configure the fix/_network for the nodes with Vlan and the nodes fail to get IPs. I was wondering if it is possible to trick the heat templates to create vxlan network instead of vlan. Regards Tony Karera

3 7

[nova][placement] Xena release
by Balazs Gibizer 06 Sep '21

06 Sep '21

Hi, We are getting close to the Xena release. So I created a tracking etherpad[1] with the schedule and TODOs. One thing that I want to highlight is that we will hit Feature Freeze on 3rd of September. As the timeframe between FF and RC1 is short I'm not plannig with FFEs. Patches that are approved before 3rd of Sept EOB can be rechecked or rebased if needed and then re-approved. If you have a patch that is really close but not approved before the deadline, and you think there are two cores that willing to review it before RC1, then please send a mail to the ML with [nova][FFE] subject prefix not later than 7th of Sept EOB. Cheers, gibi [1] https://etherpad.opendev.org/p/nova-xena-rc-potential

1 1

[cinder] discuss nas_secure options and root_squash (prohibiting root access to share)
by Stefan Hoffmann 06 Sep '21

06 Sep '21

Hi cinder team, like discussed in the last meeting, I prepared a list [1] of combinations of the nas_secure options and when to use them. If one want to prohibit root access to NFS share, only setting nas_secure_file_operations and nas_secure_file_permissions to true is a useful option, I think. (Option 4) But also the nas_secure_file_operations is not useful to determine if _qemu_img_info and fs access check at _connect_device should be done with root user or cinder user. So I will update the change [2] like proposed in the etherpad. Feel free to add other use cases and hints for the options to [1] and discuss about the proposed change. Regards Stefan [1] https://etherpad.opendev.org/p/gSotXYAZ3JfJE8FEpMpS [2] https://review.opendev.org/c/openstack/cinder/+/802882 Initial Bug: https://bugs.launchpad.net/cinder/+bug/1938196?comments=all

1 1

[kolla-ansible] Running Docker Swarm with Kuryr networking
by Paolo Celati 02 Sep '21

02 Sep '21

Hi, long story short I have a 3 node Openstack cluster that I manage with kolla-ansible, and I'd like to run Docker Swarm on that as well. I am aware Magnum exists, but I'd first like to get my head around this simpler case. Seeing as I'd like to connect Docker containers from swarm compose files to Neutron networks I'm trying to set up Kuryr together with a swarm configuration. However the documentation is a little scarce <https://docs.openstack.org/kolla-ansible/latest/reference/containers/kuryr-…> and I'd prefer running everything on these three hosts, including etcd. If I follow the guide and pass --cluster-store and --cluster-advertise arguments to dockerd then I can't run Docker in Swarm mode because I get an error saying Swarm is incompatible with those options, and at the same time it's not clear from documentation how you are expected to do Kuryr+Swarm. I did initialise the Swarm cluster before trying to add Kuryr, so I don't know if perhaps doing this the other way works? Do you have ideas or advice with this scenario? If worst comes to worst I can set up an external etcd cluster on a separate non-Openstack cluster but I'd rather avoid that. Thanks in advance, Paolo

4 9

[nova][cinder][barbican] storing secrets in service project
by Pavlo Shchelokovskyy 02 Sep '21

02 Sep '21

Hi all, When Barbican is involved, and services like Nova and Cinder create secrets behind the scenes themselves, this results in problems deleting or otherwise managing some resources by cloud admins. For example as described in https://bugs.launchpad.net/cinder/+bug/1775372, admin can not delete an encrypted volume created by another user, and needs to resort to adding herself to the project first with appropriate roles. This stems IMO from the difference in default resource access policies between Barbican and other services that consume it. For example, while in Nova or Cinder, cloud admin by default can delete servers or volumes in other projects, in Barbican by default only secret creator or admin "in the same project" can delete a secret. I came up with at least 3 solutions (already mentioned in the issue above) but would like to discuss their validity, pros and cons, and especially how they affect the security model and the use cases that this Nova-Cinder-Barbican integration aims to solve. 1. Just ignore failure to delete a secret and log appropriate warning that a secret may be left orphaned. Obviously not the best solution, but rather simple one, also will protect from cases when for whatever reason a secret will be deleted from Barbican directly (e.g. by mistake), rendering volume undeletable for anyone. 2. Change default barbican policies to allow secret deletion for cloud admin (system scope?). Naturally has implications for the Barbican model for resources access, so may be not that appropriate. 3. Store the secrets in the service project instead of user's. Currently those secrets are generated by Nova or Cinder themselves, and user quite probably is not even aware that there was something created in Barbican on her behalf. The user also has no option to pass her own secrets to any of the involved API calls, so this is something happening completely behind the scenes. So do we really need to store these secrets in the user's project? Wouldn't it be better to store them in the service project, have the same service user defined in Nova and Cinder, so it can operate and access those secrets regardless of who manages the resource which depends on access to this secret? This also may solve the problem of encrypted local instance images in Nova, where upon host restart, Nova is not able to bring those instances up because it lacks the context capable of accessing those secrets. I think the third option is the best one, but I may be missing the actual use case meant behind this feature. Is it "keep the data encrypted at rest and in transit" or more strictly "encrypt data of different users with different keys and limit access to those keys" (for the case of breach etc) ? In the first case it seems storing all the auto-generated secrets in a service project would suffice... Please help me understand the best approach to take. Best regards, Pavlo.

4 4

[tc][all]Please help to identify common pain points & video meeting scheduled next week
by Rico Lin 02 Sep '21

02 Sep '21

Dear all First, I thank Kendall and everyone who help with the Pain point elimination etherpad [1]. Two paths we plan to work on or would like to see how we can push it forward: 1. To identify common pain points from OpenStack. And see if we can target them in the following cycles. 2. To see if we can have a community-wide goal to encourage project teams to pick up their own pain point in the following cycles. *This ML is focused on the first path; To identify common pain points.* So I would like to encourage all to help to identify common pain points from it and also help put your feedbacks on the etherpad [1]. To identify common pain points, you can help to leave comments to pain points that you feel it is not an issue for a single project team but for a larger scope. We kind of recognized rabbitmq (or said RPC backend) might be one of the common pain points from that etherpad. And we wish to learn more from all of you. And we need operators' feedback mostly. Also, *we plan to put `identify common pain points` on schedule for the TC meeting next week (which is a video meeting on google meet). If you're interested, please help with etherpad, and join the TC meeting [2] next week.* [1] https://etherpad.opendev.org/p/pain-point-elimination [2] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee *Rico Lin* OIF Individual Board of directors, OpenStack TC, Multi-arch SIG chair, Heat PTL, Senior Software Engineer@EasyStack

1 1