May 2021 - openstack-discuss - lists.openstack.org

[all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community
by Liye Pang(逄立业) 10 Sep '21

10 Sep '21

Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects and that has contributed to the OpenStack community. The following is an introduction to “Venus”. If there is interest in the community, we are interested in proposing it to become an official OpenStack project in the future. Background In the day-to-day operation and maintenance of large-scale cloud platform, the following problems are encountered： l Time-consuming for log querying while the server increasing to thousands. l Difficult to retrieve logs, since there are many modules in the platform, e.g. systems service, compute, storage, network and other platform services. l The large amount and dispersion of log make faults are difficult to be discovered. l Because of distributed and interaction between components of the cloud platform, and scattered logs between components, it will take more time to locate problems. About Venus According to the key requirements of OpenStack in log storage, retrieval, analysis and so on, we introduced Venus project, a unified log management module. This module can provide a one-stop solution to log collection, cleaning, indexing, analysis, alarm, visualization, report generation and other needs, which involves helping operator or maintainer to quickly solve retrieve problems, grasp the operational health of the platform, and improve the management capabilities of the cloud platform. Additionally, this module plans to use machine learning algorithms to quickly locate IT failures and root causes, and improve operation and maintenance efficiency. Application scenario Venus played a key role in the following scenarios： l Retrieval: Provide a simple and easy-to-use way to retrieve all log and the context. l Analysis: Realize log association, field value statistics, and provide multi-scene and multi-dimensional visual analysis reports. l Alerts：Convert retrieval into active alerts to realize the error finding in massive logs. l Issue location: Establish a chain relationship and knowledge graphs to quickly locate problems. Overall structure The architecture of log management system based on Venus and elastic search is as follows: Diagram 0: Architecture of Venus venus_api: API module，provide API、rest-api service. venus_manager: Internal timing task module to realize the core functions of the log system. Current progress The current progress of the Venus project is as follows: l Collection：Develop fluentd collection tasks based on collectd to read, filter, format and send plug-ins for OpenStack, operating systems, and platform services, etc. l Index：Dealing with multi-dimensional index data in elasticsearch, and provide more concise and comprehensive authentication interface to return query results. l Analysis：Analyzing and display the related module errors, Mariadb connection errors, and Rabbitmq connection errors. l Alerts：Develop alarm task code to set threshold for the number of error logs of different modules at different times, and provides alarm services and notification services. l Location：Develop the call chain analysis function based on global_requested series, which can show the execution sequence, time and error information, etc., and provide the export operation. l Management：Develop configuration management functions in the log system, such as alarm threshold setting, timing task management, and log saving time setting, etc. Application examples Two examples of Venus application scenarios are as follows. 1. The virtual machine creation operation was performed on the cloud platform and it was found that the virtual machine was not created successfully. First, we can find the request id of the operation and jump to the virtual machine creation call chain page. Then, we can query the calling process, view and download the details of the log of the call. 2. In the cloud platform, the error log of each module can be converted into alarms to remind the users. Further, we can retrieve the details of the error log and error log statistics. Next step The next step of the Venus project is as follows: * Collection：In addition to fluent, other collection plugins such as logstash will be integrated. * Analysis: Explore more operation and maintenance scenarios, and conduct statistical analysis and alarm on key data. * display: The configuration, analysis and alarm of Venus will be integrated into horizon in the form of plugin. * location: Form clustering log and construct knowledge map, and integrate algorithm class library to locate the root cause of the fault. Venus Project Registry Venus library: https://opendev.org/inspur/venus You can grab the source code using the following git command: git clone https://opendev.org/inspur/venus.git Venus Demo Youtu.be: <https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM

7 15

[interop][cinder] review of next.json (part 2)
by Brian Rosmaita 09 Aug '21

09 Aug '21

Hello Interop WG, The Cinder team completed its discussion of what capabilities should be added as advisory in next.json at yesterday's weekly cinder meeting [0]. The tldr; is that we don't have anything to propose for inclusion in next.json at the present time. The team does have some general questions that would help us determine the suitability of some proposed capabilities. We'd like to invite someone from the Interop WG to the cinder xena R-9 virtual midcycle (Wednesday 4 August 2021, 1400-1600 UTC) so we can discuss this "live". So if someone could put 1400 UTC 4 August on their schedule for a 20 minute discussion, that would be very helpful. cheers, brian [0] http://eavesdrop.openstack.org/meetings/cinder/2021/cinder.2021-05-12-14.00…

3 8

placement service issue (placement-status upgrade check failure)
by Deepak Mohanty 12 Jul '21

12 Jul '21

I need your help with installation of placement service on Ubuntu 20.04. I have followed the steps in: https://docs.openstack.org/placement/wallaby/install/install-ubuntu.html I am facing the following issue when I try to verify the installation: $ placement-status upgrade check Error: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/oslo_upgradecheck/upgradecheck.py", line 196, in run return conf.command.action_fn() File "/usr/lib/python3/dist-packages/oslo_upgradecheck/upgradecheck.py", line 104, in check result = func_name(self, **kwargs) File "/usr/lib/python3/dist-packages/oslo_upgradecheck/common_checks.py", line 41, in check_policy_json policy_path = conf.find_file(conf.oslo_policy.policy_file) File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2543, in find_file raise NotInitializedError() oslo_config.cfg.NotInitializedError: call expression on parser has not been invoked I did a bit of debugging and found that the policy file is: policy.json. I could not find any policy.json on my machine. I created an empty policy.json at: /etc/placement/policy.json (user and group = placement). That too did not work. I have followed the steps in the install guide twice, deleting the placement database and other resources and recreating them. I continue to see the same issue. Thank you for your time.

5 6

[all][infra][qa] Retiring Logstash, Elasticsearch, subunit2sql, and Health
by Clark Boylan 09 Jul '21

09 Jul '21

Hello everyone, Xenial has recently reached the end of its life. Our logstash+kibana+elasticsearch and subunit2sql+health data crunching services all run on Xenial. Even without the distro platform EOL concerns these services are growing old and haven't received the care they need to keep running reliably. Additionally these services represent a large portion of our resource consumption: * 6 x 16 vcpu + 60GB RAM + 1TB disk Elasticsearch servers * 20 x 4 vcpu + 4GB RAM logstash-worker servers * 1 x 2 vcpu + 2GB RAM logstash/kibana central server * 2 x 8 vcpu + 8GB RAM subunit-worker servers * 64GB RAM + 500GB disk subunit2sql trove db server * 1 x 4 vcpu + 4GB RAM health server To put things in perspective, they account for more than a quarter of our control plane servers, occupying over a third of our block storage and in excess of half the total memory footprint. The OpenDev/OpenStack Infra team(s) don't seem to have the time available currently to do the major lifting required to bring these services up to date. I would like to propose that we simply turn them off. All of these services operate off of public data that will not be going away (specifically job log content). If others are interested in taking this on they can hook into this data and run their own processing pipelines. I am sure not everyone will be happy with this proposal. I get it. I came up with the idea for the elasticsearch job log processing way back at the San Diego summit. I spent many many many hours since working to get it up and running and to keep it running. But pragmatism means that my efforts and the team's efforts are better spent elsewhere. I am happy to hear feedback on this. Thank you for your time. Clark

10 13

[Octavia dashboard] Impossible to fill the subnet field when creating a load balancer
by Taltavull Jean-Francois 09 Jul '21

09 Jul '21

Hi All, On my OSA Victoria deployment (on Ubuntu 20.04), the "Create Load Balancer" form does not allow me to fill the mandatory "subnet" field and therefore I can't create a load balancer with Horizon. On the other hand, Octavia works fine and I can create load balancers with the CLI, Terraform, etc. Has one of you already faced such a situation ? Best regards, Jean-François

3 6

[nova] spec review day
by Balazs Gibizer 05 Jul '21

05 Jul '21

Hi, During the today's meeting it came up that we should have a spec review day around M1 (and later one more before M2). As M1 is 27th of May, I propose to do a spec review day on 25th of May, which is a Tuesday. Let me know if the timing does not good for you. The rules are the usual. Let's use this day to focus on open specs, trying to reach agreement on as many thing as possible with close cooperation during the day. Cheers, gibi

1 3

[Swift] Object replication failures on newly upgraded servers
by Mark Kirkwood 01 Jul '21

01 Jul '21

HI, I'm in the process of upgrading a Swift cluster from 2.7/Mitaka to 2.23/Train. While in general it seems to be going well, I'm noticing non-zero object replication failures on the upgraded nodes only, e.g: $ curl http://localhost:6000/recon/replication/object {"replication_last": 1622156911.019487, "replication_stats": {"rsync": 40580, "success": 4141229, "attempted": 2081856, "remove": 4083, "suffix_count": 14960481, "failure": 26550, "hashmatch": 4127197, "failure_nodes": {"10.11.18.67": {"obj08": 2348, "obj09": 60, "obj10": 3030, "obj02": 34, "obj03": 25, "obj01": 44, "obj06": 1498, "obj07": 28, "obj04": 69, "obj05": 36}, "10.11.18.68": {"obj03": 6901, "obj01": 293, "obj06": 1901, "obj04": 10281, "obj10": 1}, "10.12.18.76": {"obj10": 1}}, "suffix_sync": 1785, "suffix_hash": 2778}, "object_replication_last": 1622156911.019487, "replication_time": 1094.7836411476135, "object_replication_time": 1094.7836411476135} Examining the logs (/var/log/swift/object.log and /var/log/syslog) these are not throwing up any red flags (i.e no failing rsyncs noted). Any suggesting about how to get more information about what went wrong e.g: "10.11.18.67": {"obj08": 2348}, how to find what those 2348 errors were? regards Mark P.s: basic sanity checking is ok - uploaded objects go where they should and can be retrieved for 2.7 or 2.23 servers ok (the old and new version servers agree about object placement)

2 4

[tc][all] What should we have as community goals for Y series?( Starting community-wide goals ideas)
by Rico Lin 30 Jun '21

30 Jun '21

Dear all, We're now in R-22 week for Xena cycle which sounds like a perfect time to start calling for community-wide goals ideas for Y-series. According to the goal process schedule [1], we need to find potential goals, and champions before Xena milestone-1 and provide proper discussion in the community right after that to give a clear view and detail on each goal. And if we would like to keep up with the schedule, we should start right away to identify potential goals. So please help to provide ideas for Y series community-wide goals in [2]. Community-wide goals are important in terms of solving and improving a technical area across OpenStack as a whole. It has a lot more benefits to be considered from users as well from a developer's perspective. See [3] for more details about community-wide goals and processes. Also, you can refer to the backlogs of community-wide goals from this[4] and victoria cycle goals[5] (also ussuri[6]). We took cool-down cycle goal step for Xena cycle [7], so no selected goals for Xena. [1] https://governance.openstack.org/tc/goals/#goal-selection-schedule [2] https://etherpad.opendev.org/p/y-series-goals [3] https://governance.openstack.org/tc/goals/index.html [4] https://etherpad.openstack.org/p/community-goals [5] https://etherpad.openstack.org/p/YVR-v-series-goals [6] https://etherpad.openstack.org/p/PVG-u-series-goals [7] https://review.opendev.org/c/openstack/governance/+/770616 *Rico Lin* OIF Board director, OpenStack TC, Multi-arch SIG chair, Heat PTL, Senior Software Engineer@EasyStack

10 13

Customization of nova-scheduler
by Levon Melikbekjan 16 Jun '21

16 Jun '21

Hello Openstack team, is it possible to customize the nova-scheduler via Python? If yes, how? Best regards Levon

8 16

[all] CRITICAL: Upcoming changes to the OpenStack Community IRC this weekend
by Ghanshyam Mann 15 Jun '21

15 Jun '21

Greetings contributors & community members! With recent events, the Technical Committee held an emergency meeting today (Wednesday, May 26th, 2021) regarding Freenode IRC and what our decision would be [1]. Earlier in the week, the consensus amongst the TC was to gather more information from the individual projects, and make a decision from there[2]. With #rdo, #ubuntu, and #wikipedia having been hijacked, the consensus amongst the TC and the community members who were able to attend the meeting was to move away from Freenode as soon as possible. The TC agreed that this move away from Freenode needs to be a community-wide move to the same, new IRC network for all projects to avoid splintering of the community. As has been long-planned in the event of a contingency, we will be moving to OFTC. We recognize this is a contentious topic, and ultimately we seek to ensure community continuity before evolution to something beyond IRC, as many have expressed interest in doing via Mailing List discussions. At this point, we had to make a decision to solve the immediate problem in the simplest and most expedient way possible, so this is that announcement. We welcome continued discussion about future alternatives on the other threads. With this in mind, we suggest the following steps. Everyone: ======= 1. Do NOT change any channel topics to represent this change. This is likely to result in the channel being taken over by Freenode and will disrupt communications within our community. 2. Register your nicknames on OFTC [3][4] 3. Be *prepared* to join your channels on OFTC[4]. The OpenStack community channels have already been registered on OFTC and await you. 4. Continue to use Freenode for OpenStack discussions until the bots have been moved and the official cut-over takes place this coming weekend. We anticipate using OFTC starting Monday, May 31st. Projects/Project Leaders: ==================== 1. Projects should work to get a few volunteers to staff their project channels on Freenode, for the near future to help redirect people to OFTC. This should occur via private messages to avoid a ban. 2. Continue to hold project meetings on Freenode until the bots are enabled on OFTC. 3. Update project wikis/documentation with the new IRC network information. We ask that you consider referring to the central contributor guide[5]. 4. The TC is asking that projects take advantage of this time of change to consider moving project meetings from the #openstack-meeting* channels to their project channel. 5. Please avoid discussing the move to OFTC in Freenode channels as this may also trigger a takeover of the channel. We are working on getting our bots over to OFTC, and they will be moved over the weekend. Starting Monday May 31, the bots will be on OFTC. Communication regarding this migration will take place on OFTC[4] in #openstack-dev, and we're working on updating the contributor guide[5] to reflect this migration. Sincerely, The OpenStack TC and community leaders who came together to agree on a path forward. [1]: https://etherpad.opendev.org/p/openstack-irc [2]: https://etherpad.opendev.org/p/feedback-on-freenode [3]: https://www.oftc.net/Services/#register-your-account [4]: https://www.oftc.net/ [5]: https://docs.openstack.org/contributors/common/irc.html

3 6